We are not using the URL


German / Google Webmaster Central Office hours in English from 17 October 2019

that is not in the


noindex.


The search console


is a


canonical HTTP


site.


So,


how do we know


what we are


using?


We are


not using


the URL for the


indexing.


What happens


is that we


use the JavaScript


URL for the


rendering.


And if we


use the


server-level URL for the


rendering, then we


use the resource URL


for the indexing.


If I use


the JavaScript URL


for the


normal HTML site,


then this


will be separate from the


text that we


used to index.


So we make it


normal for the HTML links


to be used in the


in-house.


We will not use


the server logs that we


used in the data.


We will not use the rendering


node.


But we do not use the index


for the rendering.


If you have not used the


server logs,


you can use the


website URL


and you will not use the


URL links for the


data.


But the normal URL will


not be used for the indexing.


If you


use the


URL that you


use and you do not use the


URL, you can use


the X-Robots tag.


This is


a hardtpt header.


You can use it


in different ways and then


you can use the NoIndex.


And then when


these data are separated


from the indexing,


then you will not use the


JavaScript data.


You will not use the


indexing.


You can use a canonical


version of the JavaScript data.


I think that


the most important thing is not


so much that you


use a JavaScript data,


then it is not clear what the canonical


version of the JavaScript data


is.


The only reason is that


the NoIndex


is not used for the JavaScript data.


You will not use


the separate indexing.


I would like to


use the separate logs, but you will not


use the separate indexing.


We have


seen a lot of


the site


by Johannes


that I have


already said.


The name of my


name is


the one that


the moment that


the data is not in the index.


The JavaScript or the


JavaScript data


is not used.


But the name


in the console is the


scrolling output.


That is why I


can use the same


information and the scrolling allows.


In my case, I have already said


that the entire resource


cannot be crawled.


My idea was to


use the


indexing.


That is why I had


to pay


a NoIndex


implementer.


That is not


true for the content fragment.


That is the problem.


That is the


only reason that


these fragments


are not used in the


site. That is not


true.


The NoIndex


is not used in the URL.


That is why


the URL for the index


is not used.


If the URL for the index is not used


in the URL,


then the URL for the


style is not used in the site.


That is not true.


Ah, super.


I understand.


Super.


We have


a few minutes left.


What do you think about


the URL for the


site?


One thing


that was new for me.


I was just


in the room and I


experimented with the dash and


the search and I was also


working on it.


The title


of the URL is also in the breadcrumb.


That is why I


am working on it.


You can find it in the site.


You can find it in the website.


I think that is what it is.


I am


really excited to see


how the site works.


You may not know how to experiment.


You can


find something good.


You can find something good.


But it is


something that we say


we want to do and


we are looking for something


that is good for us.


That is why


we are working on


the RBA test.


We are working on it.


The title


of the site


is in the website.


I have not


gotten any feedback from


the SEOs.


They are not the ones


that are really good.


They are the URLs.


I am also


looking for the mobile


site.


I think that is consistent.


I am looking for the mobile


But on the desktop


I am looking for


control F and


websites that are


really good.


I think that is


good for the


website.


The feedback


that we are


creating on Twitter is


really good on the team.


But it is also


important that the


SEOs are not


really good for the


The SEOs are not


The SEOs are not really good for the


The normal website


is not really good for the website.


But that is


something that we are


working on on the team.


We are still working on


the URL.


I think


that is good.


I think the parallel


between mobile and


tech is good.


The third is good for the


website. I think that mobile has


breadcrumb and desktop ones


are also good for the website.


We can also do


that as well.


For me it is a bit


clicky-bunty but


it is really good.


I have no problem with that.


I am not sure


how to work on the


website.


I am just


curious about the


experience.


It is a bit


confusing.


I think that in


a big, big, big


way we can


do something new.


I think that this


will be a big


opportunity for us to


try to do something


new.


If you have not already.


I am not sure what to say.


No?


Ok.


I think


I am going to go on.


I think that Hans is going to do


something new.


Hans, if you want to


Ok, let's go.


This is the


automatic resource.


How do you


use it with


PRP data?


It is


the source.


The source is functional


and it is not the resource


itself.


Is that


the case?


No, not really.


We will do that.


Yes, we are not too


elastic.


The direct-to-the-way can


also be used


and then the elastic


point PRP


that


is normal


and can be used


as a function


is also used


with the state-of-the-art document.


But the only


way to do it is


now.


So this PRP data can be


used in the


same way as the


other resources.


Yes.


That would not be possible.


If you implement


this so that


the PRP data


is used in the source code


for the code,


then it would not be possible.


Can you


tell us


about the


PRP data?


No, I don't.


I have to say that.


Okay, thanks.


Cool.


Thank you for your


support.


The next


Hangout is


next.


You can find the


data for the next one.


Then we will


take a short break.


And we will see you next time.


URL Inspection Tool - Google Search Console Training

Hi, I'm Daniel Weisberg, Search Advocate at Google.


And in this video, I'll talk about Search Console's URL Inspection Tool,


which provides information about Google's indexed and live versions


of a specific page on your site.


This is one of the most important debugging tools to help you understand


why your page might not be appearing the way you expect in search results.


By the end of this video, you should be able to find out


the current index status of your pages, test a live URL,


ask Google to crawl a specific page, and view detailed information


about the page's loaded resources and other information.


Google Analytics


You can access this tool from the top bar, from the navigation sidebar,


or anywhere you see a little magnifying glass next to a URL in a table.


Paste the full URL you want to check into the Inspect Search box


in Search Console, and press Enter.


To use the URL Inspection Tool, the URL you're inspecting


must be in the current property.


Note that this tool can only be used for web pages


and will only show results for web search.


Once the page displays the results, you will see three different sections,


all of them presenting information from Google's last crawl,


or crawl attempt, not from a live test of the page.


In the presence on Google Card, you'll get a verdict on whether or not


the URL can appear in Google search results.


For example, your URL can return a response that the URL is on Google,


which means you're good.


Or you might learn that the URL is on Google but has issues,


which means the URL has been indexed and can appear in Google search results,


but there might be issues with structured data in the page.


In the Covert section, you'll learn where the page was discovered,


such as a sitemap or a referring page,


when was the last crawl and by which user agent,


and whether the page is included in the Google Web Index,


or maybe another version of it was chosen as the canonical


for the Google index.


For example, here you might discover that a specific page


is currently submitted in your sitemap but marked as no index,


or that there is a server error on your side


and Google couldn't reach a number of pages.


In the Enhancement section, you'll find any structured data details,


along with AMP and mobile usability warnings and errors.


For example, if your page is not marked up properly with structured data,


the inspection will return an error detailing the missing or wrong values.


You can learn more about each of the sections


in the Search Console Help Center.


Once you get the current page status,


you'll have three main actions to perform.


First, you can click Test Live URL to see if the page can be indexed by Google.


This runs a test against the live page for information similar to the indexed URL.


This is very useful to test live if an issue still exists,


even after you've fixed it in your site.


If you want to see a live rendered version of your page by Google,


this is the only place you'll find it.


This test confirms whether Googlebot can access your page for indexing.


However, even if a live test is positive,


your page must still fulfill other conditions in order to be indexed,


such as following the Webmaster Guidelines.


Second, you can click Request Indexing for Google to re-index your page


if you make changes to it.


For example, if you fixed a structured data error,


enhanced your mobile usability,


or performed a major change in the page,


you might want to request indexing.


And third, you can view more information about the tested page


by clicking View Crawled Page.


This will show the retrieved HTML


and more info about the HTTP response and page resources loaded.


Wow, that really is interesting.


I hope now you understand how to inspect a URL


to get information about Google's indexed version


of a specific page on your site.


That might come really handy when analyzing your search efforts on Google.


In the next episode,


I'll talk about how to monitor your search appearance,


where to find errors in your structured data,


and how to ask Google to validate fixes you've done to your site.


Don't forget to subscribe to the Google Webmasters YouTube channel,


where we'll be publishing lots of Search Console videos.


So stay tuned.


Google Webmasters


Deliver search-friendly JavaScript-powered websites (Google I/O '18)


Good morning, everyone. My name is Tom Greenway and I'm a


partner developer advocate from Google Sydney with a focus on


the indexability of progressive web applications.


Hi, everyone. I'm John Mueller. I'm a webmaster trends analyst


from Zurich in Switzerland. It's great to see so many of you


here, even at this early hour.


I'm going to talk about the web page.


The web page is a web page that's


a web page that's a web page that's a web page.


The web page is a web page that's a web page that's a web


page.


It's a web page that's a web page that's a web page.


Now, as you can imagine, John and I have a lot of experience


with the work that web developers must do to ensure


that websites are indexable which is another way of saying


that websites are indexable.


We have a lot of experience with web


developers and web developers and web


developers and web developers and web developers and web


developers.


I'm excited to tell you that in this


talk, we're announcing a bunch of cool new stuff, including a


new change to Google Search policy, a new approach for


rendering HTML to search crawlers and even a new Google


Assistant series for a


business of open web service.


Join Weird Cat today.


making an image safe to do.


These are really critical questions.


And as developers ourselves, we understand the frustration


behind this mystery.


So today, John and I are going to do something


we very rarely do at Google.


We're going to pull back the curtain a little bit


and reveal some new pieces of information about how


Google Search sees the web and indexes it.


And with this new knowledge and a few new tools,


you'll have concrete steps you can


take to ensure the JavaScript-powered websites


you're building are visible to Google Search.


Now, I want to remind you that this talk is


about modern JavaScript-powered websites.


And typically, these websites will be powered by JavaScript


frameworks, such as Angular, Polymer, React, or Vue.js.


And who doesn't love a great web development framework that's


easy to use and helps you build your sites faster


and works great for your users?


But it's important to recognize that some of these frameworks


use a single-page app configuration model,


meaning they use a single HTML file that


pulls in a bunch of JavaScript.


And that can make a lot of stuff simpler.


But if you don't watch out, JavaScript-powered websites


can be a problem for search engines.


So let's take a quick look at what


the default template for a new Angular project looks like.


As you can see, the default project template is pretty


basic.


It shows you how to use Angular to render a header, an image,


and a few links.


Nice and simple.


How could this possibly be a problem from an indexability


perspective?


Well, let's take a peek behind the scenes at the HTML.


This is it.


Take a good look.


When viewed in the browser, the default sample project


had text, imagery, and links.


But you wouldn't know that from looking at this initial HTML


that's been delivered from the server, now would you?


The initial HTML that's been sent down


is actually completely devoid of any content.


See here in the app root?


That's all there is in the body of the page


except for some script tags.


So some search engines might assume that there's actually


nothing here to index.


And to be clear, Angular isn't the only web framework


that serves an empty response on its initial server side render.


Polymer, React, and vue.js have similar issues by default.


So what does this mean for the indexability of our websites


from the perspective of Google Search?


Well, to answer that question better,


we'll take a little step back and talk about the web


in general, why search engines exist,


and why search crawlers are necessary.


Perhaps a good question to start with is, how big is the web?


Well, we can tell you that we've actually found over 130


trillion documents on the web.


So in other words, it's really big.


And as you know, the aim of all search engines,


including Google, is to provide a list of relevant search


results based on a user's search query.


And to make that mapping of user queries,


the search results fast and accurate,


we need an index, similar to the catalog of a gigantic library.


And given the size of the web, that's a really complex task.


And so to build this index to power our search engine,


we need another tool, a search crawler.


And traditionally, a search crawler


was basically just a computer and a piece of software


that performed two key steps.


One, it aims to find a piece of content to be crawled.


And to do this, the content must be retrievable via URL.


And once we have a URL, we get its content,


and we sift through the HTML to index the page


and find new links to crawl as well.


And thus, the cycle repeats.


So let's look at that first step, the crawling,


and break it down.


Oh, and yes, as an Australian, I felt


it was imperative that I include some spiders in my talk.


So this is the cutest possible one I could find.


John, what do you think?


No, you're not convinced?


OK.


Well, I have a few more in the deck.


So maybe you'll come around.


So to ensure the crawling is possible,


there are some key things to keep in mind.


Firstly, we need URLs to be reachable,


as in there shouldn't be any issue when the crawler wants


to request the web pages and retrieve


the resources necessary for indexing them from your web


server.


And secondly, if there are multiple documents


that contain the same content, we


need a way to identify the original source.


Otherwise, it could be interpreted


as duplicate content.


And finally, we also want our web pages to have clean,


unique URLs.


Originally, this was pretty straightforward on the web.


But then the first single page apps


made things a bit more complicated.


So let's go over each of these concepts.


First, for the reachability of URLs,


there's a simple standard way to help search engines find content


that you're probably familiar with.


You add a plain text file called robots.txt


to the top level domain of your site,


which specifies which URLs to crawl and which to ignore.


And I say URLs because these rules


can prevent JavaScript from being crawled too, which


could affect your indexability.


And this example also gives us a link to a sitemap.


A sitemap helps crawlers by providing a recommended set


of URLs to crawl initially for a site.


And to be clear, there's no guarantee


these URLs will get crawled.


They're just one of the signals that search crawlers


will consider.


OK.


But now let's talk about that duplicate content scenario


and how search crawlers deal with this situation.


Sometimes websites want multiple pages


to have the same content, right?


Even if it's a different website.


For example, bloggers will publish articles


on their website and cross-post to services like Medium


to increase the reach of that content.


And this is called content syndication.


But it's important for search crawlers to understand which


URL you prefer to have indexed.


So the canonical metadata syntax shown here in the HTML


allows the duplicate documents to communicate to crawlers


where the original authoritative source for the content lives.


We call that source document the canonical document.


And traditionally, URLs for the web started out quite simple.


Just a URL that was fetched from a server with some HTML.


But then, of course, AJAX came along


and just changed everything.


Suddenly, websites could execute JavaScript,


which could fetch new content from the server


without reloading the browser page.


But developers still wanted a way


to support back and forward browser


navigation and history as well.


So a trick was invented which leveraged something called


the fragment identifier.


And its purpose is for deep linking


into the sub-content of a page, like a subsection


of an encyclopedia article.


And because fragment identifiers were supported by browsers


for history and navigation, this meant developers


could trick the browser into fetching new content


dynamically without reloading the browser page,


and yet also support the history and the navigation we love


about the web.


But we realized that using the fragment identifier for two


purposes, subsections on pages and also deep linking


into content, it wasn't very elegant.


So we moved away from that.


And instead, another approach was


proposed to use the fragment identifier


followed by an exclamation mark, which we call the hashbang.


And this way, we could discern the difference


between a traditional URL using the fragment identifier


for the sub-content on a page versus a fragment identifier


being used by JavaScript to deep link into a page.


And this technique was recommended for a while.


However, nowadays, there is a modern JavaScript


API that makes these old techniques less necessary.


And it's called the History API.


And it's great because it enables managing


the history state of the URL without requiring


complete reloads of the browser all through JavaScript.


So we get the best of both worlds,


dynamically fetch content with clean traditional URLs.


And I can tell you that from Google's perspective,


we no longer index that single hash workaround.


And we discourage the use of the hashbang trick as well.


OK, well, that's crawling out of the way.


Now let's move on to the indexing step.


So web crawlers ideally want to be


able to find all the content on your website.


If the crawlers can't see some content,


then how are they going to index it?


And the core content of the page includes all the text, imagery,


video, and even hidden elements like structured metadata.


In other words, it's the HTML of the page.


But don't forget about that content you dynamically fetch,


either.


This could be worth indexing as well, such as Facebook


or discuss comments.


Crawlers want to see this embedded content too.


And also, this might seem really obvious,


but I want to emphasize that at Google,


we take HTTP codes pretty seriously, especially


404 not found codes.


If crawlers find a page that has a 404 status code,


then they probably won't even bother indexing it.


And lastly, of course, a crawler wants


to find all the links on a page as well,


because these links allow the crawlers to crawl further.


So now let's just talk a bit about those links quickly,


because honestly, they're some of the most important parts


of the web.


How do search crawlers like Google find links?


Well, I can't speak for all search crawlers,


but I can say that at Google, we only analyze one thing, anchor


tags with href attributes.


And that's it.


For example, this span here that I've just added,


it won't get crawled, because it's not an anchor.


And this additional span I've added,


even though it's an anchor, it doesn't have an href attribute.


But if you are using JavaScript, such as with the history API


that I mentioned earlier, to navigate the page purely


on the client and fetch new content dynamically,


you can do that, so long as you use the anchor


tags with href attributes, like in this last example.


Because most search crawlers, including Google,


will not simulate navigation of a page to find links.


Only the anchor tags will be followed for linking.


But wait, is that really everything?


In order to have sifted through the HTML to index the page,


we needed to have the HTML in the first place.


And in the early days of the web,


the server likely gave us all the HTML that was necessary.


But nowadays, that's not really the case.


So let's insert a step between crawling and indexing,


because we need to recognize that the search crawlers


themselves might need to take on this rendering task as well.


Otherwise, how will the search crawler


understand the modern JavaScript powered websites


we're building?


Because these sites are rendering their HTML


in the browser itself, using JavaScript and templating


frameworks, just like that Angular sample


I showed you earlier.


So when I say rendering, I don't mean drawing pixels


to the screen.


I'm talking about the actual construction of the HTML


itself.


And ultimately, this can only ever happen on either the server


or on the client.


Or a combination of the two could be used,


and we call that hybrid rendering.


Now, if it's all pre-rendered on the server,


then a search engine could just index that HTML immediately.


But if it's rendered on the client,


then things get a little bit trickier, right?


And so that's going to be the challenge that we'll


be discussing today.


But one last term, you might be wondering,


what is Google Search's crawler called?


Well, we call it Googlebot, and we'll


be referring it to it a lot in this talk.


And I think another detail to note


is that I said that a search crawl is basically


just a computer with some software running on it.


Well, obviously, maybe in the 90s that was the case.


But nowadays, due to just the sheer size of the web,


Googlebot is comprised of thousands of machines


running all this distributed software that's


constantly crunching data to understand


all of this continuously expanding information


on the web.


And to be honest, I think we sometimes


take for granted just how incredible Google


Search really is.


For example, I recently learned that with the Knowledge Graph,


which is a database of all the information we have on the web,


it actually maps out how more than 1 billion things


in the real world are connected and over 70 billion facts


between them.


It's kind of amazing.


OK, well, now that we know the principles of a search crawler,


let's see how these three different key steps, crawling,


rendering, and indexing, all connect.


Because one crucial thing to understand


is the cycle of how Googlebot works


or how it should ideally work.


As you can see, we want these three steps to hand over


to one another instantly.


And as soon as the content is fully rendered,


we want to index it to keep the Google Search index as


fresh as possible.


This sounds simple, right?


Well, it would be if all the content was


rendered on the server and complete when we crawl it.


But as you know, if a site uses client-side rendering,


then that's not going to be the case,


just like that Angular sample I showed you earlier.


So what does Googlebot do in this situation?


Well, Googlebot includes its own renderer,


which is able to run when it encounters pages


with JavaScript.


But rendering pages at the scale of the web


requires a lot of time and computational resources.


And make no mistake, this is a serious challenge


for search crawlers, Googlebot included.


And so we come to the important truth


about Google Search we would like to share with you today,


which is that currently, the rendering of JavaScript-powered


websites in Google Search is actually


deferred until Googlebot has the resources available to process


that content.


Now, you might be thinking, OK, well, what does that really


mean?


Well, I'll show you.


In reality, Googlebot's process looks a bit different.


We crawl a page, we fetch the server-side rendered content,


and then we run some initial indexing on that document.


But rendering the JavaScript-powered web pages


takes processing power and memory.


And while Googlebot is very, very powerful,


it doesn't have infinite resources.


So if the page has JavaScript in it,


the rendering is actually deferred


until we have the resources ready to render


the client-side content.


And then we index the content further.


So Googlebot might index a page before rendering is complete,


and the final render can actually


arrive several days later.


And when that final render does arrive,


then we perform another wave of indexing


on that client-side rendered content.


And this effectively means that if your site


is using a heavy amount of client-side JavaScript


for rendering, you could be tripped up at times


when your content is being indexed


due to the nature of this two-phase indexing process.


And so ultimately, what I'm really trying to say


is because Googlebot actually runs


two waves of indexing across your content,


it's possible some details might be missed.


For example, if your site is a progressive web application


and you've built it around a single-page app model,


then it's likely all your unique URLs


share some base template of resources


which are then filled in with content by Ajax or fetch


requests.


And if that's the case, consider this.


Did the initially server-side rendered version of the page


have the correct canonical URL included in it?


Because if you're relying on that to be rendered by the client


then we'll actually completely miss it


because our second wave of indexing


doesn't check for the canonical tag at all.


Additionally, if the user requests a URL that


doesn't exist and you attempt to use JavaScript


to send the user a 404 page, then we're


actually going to miss that too.


Now, John will talk more about these issues later in the talk.


But the important thing to take away right now


is that these really aren't minor issues.


These are real issues that could affect your indexability.


Metadata, canonical tags, HTTP codes,


as I mentioned at the beginning of this talk,


these are all really key to how search crawlers understand


the content on your web pages.


However, just to be clear, not all web pages on a website


necessarily need to be indexed.


For example, actually on the Google IO schedule website,


there is a listing and filter interface for the sessions.


And we want search crawlers to find the individual session


pages, but we discovered the client-side rendered deep links


weren't being indexed because the canonical tags were


rendered in the client and the URLs were


fragment identifier based.


So we implemented a new template with clean URLs


and server-side rendered canonical tags


to ensure the session descriptions were properly


indexed because we care about that content.


And to ensure these documents were crawlable,


we added them to the site map as well.


But what about the single page app which


allows for filtering sessions?


Well, that's more of a tool than a piece of content, right?


Therefore, it's not as important to index the HTML on that page.


So ask yourself this.


Do the pages I care about from the perspective


of content and indexing use client-side rendering anyway?


OK, so now you know.


When building a client-side rendered website,


you must tread carefully.


As the web and the industry has gone bigger,


so too have the teams and companies become more complex.


We now work in a world where the people building websites


aren't necessarily the same people promoting or marketing


those websites.


And so this challenge is one that we're all


facing together as an industry, both from Google's perspective


and yours as developers.


Because after all, you want your content indexed by search


engines, and so do we.


Well, this seems like a good opportunity to change tracks.


So John, do you want to take over and tell everyone


about the Google search policy changes


and some of the best practices they can apply so we can


meet this challenge together?


Sure.


Thanks, Tom.


That was a great summary of how search works.


Though I still don't know about those pictures of spiders.


Kind of scary.


But Googlebot in reality is actually quite friendly.


Anyway, as Tom mentioned, the indexing


of modern JavaScript-powered websites is a challenge.


It's a challenge both for Google as a search engine


and for you all as developers of the modern web.


And while developments on our side are still ongoing,


we'd like to help you to tackle this challenge in a more


systematic way.


So for that, we'll look at three things here.


The policy change that we mentioned briefly before,


some new tools that are available to help


you diagnose these issues a little bit better,


and lastly, a bunch of best practices


to help you to make better JavaScript-powered websites


that also work well in search.


So we've talked about client-side rendering briefly


and server-side rendering already.


Client-side rendering is a traditional state


where JavaScript is processed on the client.


That would be the user's browser or on a search engine.


For server-side rendering, the server, so your server,


will process the JavaScript and serve mostly static HTML


to search engines.


Often, this also has speed advantages.


So especially on lower-end devices, on mobile devices,


JavaScript can take a bit of time to run.


So this is a good practice.


For both of these, we index the state as ultimately


seen in the browser.


So that's what we pick up.


And we try to render pages when we need to do that.


There's a third type of rendering


that we've talked about in the past.


It starts in the same way in that pre-rendered HTML


is sent to the client.


So you have the same speed advantages there.


However, on interaction or after the initial page load,


the server adds JavaScript on top of that.


And as with server-side rendering,


our job as a search engine is pretty easy here.


We just pick up the pre-rendered HTML content.


We call this hybrid rendering.


This is actually our long-term recommendation.


We think this is probably where things


will end up in the long run.


However, in practice, implementing this


can still be a bit tricky.


And most frameworks don't make this easy.


A quick call out to Angular.


Since we featured them in the beginning


as an example of a page that was hard to pick up,


they have built a hybrid rendering mode


with Angular Universal that helps you to do


this a little bit easier.


Over time, I imagine more frameworks


will have something similar to make it easier for you


to do this in practice.


However, at least at the moment, if your server


isn't written in JavaScript, you're


going to be dealing with double maintenance of controlling


and templating logic as well.


So what's another option?


What's another way that JavaScript sites could


work well with search?


We have another option that we'd like to introduce.


We call it dynamic rendering.


In a nutshell, dynamic rendering is the principle


of sending normal client-side rendered content to users


and sending fully server-side rendered content


to search engines and to other crawlers that need it.


This is the policy change that we talked about before.


So we call it dynamic because your site dynamically


detects whether or not the request there


is a search engine crawler, like Googlebot,


and only then sends the server-side rendered content


directly to the client.


You can include other web services here as well


that can't deal with rendering.


For example, maybe social media services or chat services.


Anything that tries to extract structured information


from your pages.


And for all other requesters, so your normal users,


you would serve your normal hybrid or client-side rendered


code.


This also gives you the best of both worlds


and makes it easy for you to migrate


to hybrid rendering for your users over time as well.


One thing to note, this is not a requirement for JavaScript


sites to be indexed.


As you'll see later, Googlebot can render most pages already.


For dynamic rendering, our recommendation


is to add a new tool or step in your server infrastructure


to act as a dynamic renderer.


This reads your normal client-side content


and sends a pre-rendered version to search engine crawlers.


So how might you implement that?


We have two options here that help


you to kind of get started.


The first is Puppeteer, which is a Node.js library, which


wraps a headless version of Google Chrome underneath.


This allows you to render pages on your own.


Another option is RenderTron, which


you can run as a software as a service that renders and caches


your content on your site as well.


Both of these are open source, so you


could make your own version or use something


from a third party that does something similar as well.


For more information on these, I'd


recommend checking out the IOs session on Headless Chrome.


I believe there's a recording about that already.


Either way, keep in mind rendering


can be pretty resource intensive.


So we recommend doing this out of band


from your normal web server and implementing caching


as you need it.


So let's take a quick look at what your server infrastructure


might look like with a dynamic renderer integrated.


Requests from Googlebot come in on the side here.


They're sent to your normal server.


And then perhaps through a reverse proxy,


they're sent to the dynamic renderer.


There, it requests and renders the complete final page


and sends that back to the search engines.


So without needing to implement or maintain any new code,


this setup could enable a website that's


designed only for client side rendering


to perform dynamic rendering of the content to Googlebot


and to other appropriate clients.


If you think about it, this kind of solves the problems


that Tom mentioned before.


And now we can be kind of confident


that the important content of our web pages


is available to Googlebot when it performs


its initial wave of indexing.


So how might you recognize Googlebot requests?


This is actually pretty easy.


So the easiest way to do that is to find Googlebot


in the user agent stream.


You can do something similar for other services


that you want to serve pre-rendered content to.


And for Googlebot, as well as some others,


you can also do a reverse DNS lookup


if you want to be sure that you're serving it just


to legitimate clients.


One thing to kind of watch out for here


is that if you serve adapted content to smartphone


users versus desktop users, or you redirect users


to different URLs depending on the device that they use,


you must make sure that dynamic rendering also returns


device-focused content.


In other words, mobile search engine crawlers,


when they go to your web pages, they


should see the mobile version of the page,


and the others should see the desktop version.


If you're using responsive design,


so if you're using the same HTML and just using CSS


to conditionally change the way the content is shown to users,


this is one thing you don't need to watch out for,


because the HTML is exactly the same.


What's not immediately clear from the user agents


is that Googlebot is currently using a somewhat older browser


to render pages.


It uses Chrome 41, which was released in 2015.


The most visible implication for developers


is that newer JavaScript versions and coding


conventions like arrow functions aren't supported by Googlebot.


And with that, also any API that was added after Chrome 41


currently isn't supported.


You can check these on a site like Can I Use?


And while you could theoretically


install an older version of Chromium,


we don't recommend doing that for obvious security reasons.


Additionally, there are some APIs


that Googlebot doesn't support because they don't provide


additional value for search.


We'll check these out soon.


All right.


So you might be thinking, this sounds like a lot of work.


I don't know.


Do I really need to do this?


So a lot of times, Googlebot can render pages properly.


Why do I really have to watch out for this?


Well, there are a few reasons to watch out for this.


First is if your site is large and rapidly changing.


For example, if you have a news website that


has a lot of new content that keeps coming out regularly


and requires quick indexing, as Tom showed,


rendering is deferred from indexing.


So if you have a large and dynamic website,


then the new content might take a while to be indexed.


Otherwise, secondly, if you rely on modern JavaScript


functionality, for example, if you


have any libraries that can't be transpiled back to ES5,


then dynamic rendering can help you there.


And that said, we continue to recommend


using proper graceful degradation techniques


so that even older clients have access to your content.


And finally, there is a third reason


to also look into this, in particular,


if you're using social media sites,


if your site relies on sharing through social media


or through chat applications, if these services require access


to your page's content, then dynamic rendering


can help you there too.


So when might you not use dynamic rendering?


I think the main aspect here is balancing the time and effort


needed to implement and to run this with the gains that


are received.


So remember, implementation and maintenance


of dynamic rendering can use a significant amount of server


resources.


And if you see Googlebot is able to index your pages properly,


then if you're not making critical high-frequency changes


to your site, maybe you don't need to actually implement


anything special.


Most sites should be able to let Googlebot render their pages


just fine.


Like I mentioned, if Googlebot can render your pages,


they probably don't need dynamic rendering for that site.


Let's take a look at a few tools to help you figure out


what the situation is.


When diagnosing rendering, we recommend


doing so incrementally.


First, checking the raw HTTP response,


and then checking the rendered version,


either on mobile or on mobile and desktop,


if you serve different content, for example.


Let's take a quick look at these.


So looking at the raw HTTP response, one way to do that


is to use Google Search Console.


To gain access to Google Search Console


and to a few other features that they have there,


you first need to verify ownership of your website.


This is really easy to do.


There are a few ways to do that, so I'd


recommend doing that regardless of what you're working on.


Once you have your site verified,


you can use a tool called Fetch as Google, which


will show the HTTP response that was received by Googlebot,


including the response code on top


and the HTML that was provided before any rendering was done.


This is a great way to double check what is happening


on your server, especially if you're


using dynamic rendering to serve different content to Googlebot.


Once you've checked the raw response,


I recommend checking how the pages actually render.


So the tool I use for this is the mobile-friendly test.


It's a really fast way of checking


Google's rendering of a page.


As I mentioned, the name suggests


that it's made for mobile devices.


So as you might know, over time, our indexing


will be primarily focused on the mobile version of a page.


We call this mobile-first indexing.


So it's good to already start focusing on the mobile version


when you're testing rendering.


We recommend testing a few pages of each kind of page


within your website.


So for example, if you have an e-commerce site,


check the home page, some of the category pages,


and some of the detail pages.


You don't need to check every page on a whole website,


because a lot of times, the templates


will be pretty similar.


If your pages render well here, then chances


are pretty high that Googlebot can render your pages for


search as well.


One thing that's kind of a downside


here is that you just see the screenshot.


You don't see the rendered HTML here.


What's one way to check the HTML?


Well, new for IO.


I think we launched this yesterday.


We've added a way to review the HTML after rendering.


This is also in the mobile-friendly test.


It shows you what was created after rendering


with the mobile Googlebot.


Includes all of the markup for links, for images,


for structured data, any invisible elements


that might be on the page after rendering.


So what do you do if the page just doesn't render properly


at all?


We also just launched a way to get full information


about loading issues from a page as well.


On this part within the mobile-friendly test,


you can see all of the resources that were blocked by Googlebot.


So this could be JavaScript files or API responses.


A lot of times, not everything needs to be crawled,


kind of like Tom mentioned.


For example, also if you have tracking pixels on a page,


Googlebot doesn't really need to render those tracking pixels.


But if you use an API to pull in content from somewhere else,


that API endpoint is blocked by robots.txt,


then obviously we can't pull in that content at all.


An aggregate list of all of these issues


is also available in Search Console.


So when pages fail in a browser, usually I


check the developer console for more information


to see more details on exceptions.


And new for I.O., one of the most requested features


from people who make JavaScript-powered sites


for Search is also showing the console log when Googlebot


tries to render something.


This allows you to check for all kinds of JavaScript issues.


For example, if you're using ES6 or if you just


have other issues with the JavaScript when it tries to run.


This makes my life so much easier


because I don't have to help people


with all of these detailed rendering issues that much.


Desktop is also a topic that still comes up.


As you've seen in maybe some of the other sessions,


desktop isn't quite dead.


So you can run all of these diagnostics in the rich results


test as well.


This tool shows the desktop version of these pages.


So now that we've seen how to diagnose issues,


what kind of issues have we run across


with modern JavaScript-powered sites?


What patterns do you need to watch out for and handle well


on your side?


So remember Tom mentioned at the beginning of the talk something


about lazy loading images and being


unsure if they're indexable?


Well, it turns out they're only sometimes indexable.


So it was good to look at that.


Depending on how lazy loading is implemented,


Googlebot may be able to trigger it.


And with that, may be able to pick up


these images for indexing.


For example, if the images are above the fold


and you're lazy loading kind of runs those images


automatically, then Googlebot will probably see that.


However, if you want to be sure that Googlebot


is able to pick up lazy loaded images, one way


to do that is to use a no script tag.


So you can add a no script tag around a normal image element,


and we'll be able to pick that up for image search directly.


Another approach is to use structured data on a page.


When we see structured data that refers to an image,


we can also pick that up for image search.


As a side note for images, we don't index images


that are referenced only through CSS.


We currently only index images that


are kind of embedded with the structured data


markup or with image tags.


Apart from lazy loaded images, there


are other types of content that require some kind of interaction


to be loaded.


What about tabs that load the content after you click on them


or if you have infinite scroll patterns on a site?


Googlebot generally won't interact with a page,


so it wouldn't be able to see these.


There are two ways that you can get this to Googlebot, though.


Either you can preload the content


and just use CSS to toggle visibility on and off.


That way, Googlebot can see that content


from the preloaded version.


Or alternately, you can just use separate URLs


and navigate the user and Googlebot to those pages


individually.


Now, Googlebot is a patient bot, but there are a lot of pages


that we have to crawl.


So we have to be efficient and kind of go


through pages fairly quickly.


When pages are slow to load or render,


Googlebot might miss some of the rendered content.


And since embedded resources are aggressively cached for search,


rendering timeouts are really hard to test for.


So to limit these problems, we recommend


making performant and efficient web pages,


which you're hopefully already doing for users anyway, right?


Anyway, in particular, limit the number of embedded resources


and avoid artificial delays like timed interstitials like here.


You can test pages with the usual set of tools


and roughly test rendering with the mobile-friendly testing


tool.


And while timeouts here are a little bit


different for indexing, in general,


if the pages work in the mobile-friendly test,


they'll work for search indexing too.


Additionally, Googlebot wants to see the page


as a new user would see it.


So we crawl and render pages in a stateless way.


Any API that tries to store something locally


would not be supported.


So if you use any of these technologies,


make sure to use graceful degradation techniques


to allow anyone to view your pages,


even if these APIs are not supported.


And that was it with regards to critical best practices.


Now it's time to take a quick circle back


and see what we see.


So first, we recommend checking for proper implementation


of best practices that we talked about.


In particular, lazy loaded images are really common.


Second, test a sample of your pages


with the mobile-friendly test.


And use the other testing tools as well.


Remember, you don't need to test all of your pages.


Just make sure that you have all of the templates covered.


And then finally, if sites are large and quick changing,


or you can't reasonably fix rendering across a site,


then maybe consider using dynamic rendering techniques


to serve Googlebot and other crawlers a pre-rendered version


of your page.


And finally, if you do decide to use dynamic rendering,


make sure to double check the results there as well.


One thing to keep in mind, indexing


isn't the same as ranking.


But generally speaking, pages do need


to be indexed before their content can appear in search


at all.


I don't know.


Tom, do you think that covers about everything?


Well, it was a lot to take in, John.


There was some amazing content.


But I guess one question I have, and I


think maybe other people in the audience


have this on their mind as well, is it always


going to be this way, John?


That's a great question, Tom.


I think things will never stay the same.


So as you mentioned in the beginning,


this is a challenge for us that's important.


Within Google Search, we want our search results


to reflect the web as it is, regardless


of the type of website that's used.


So our long-term version is that you, the developers,


shouldn't need to worry as much about this for search crawlers.


So circling back on the diagram that Tom


showed in the beginning with deferred rendering,


one change we want to make is to move rendering closer


to crawling and indexing.


Another change we want to make is


to make Googlebot use a more modern version of Chrome


over time.


Both of these will take a bit of time.


I don't like making long-term predictions,


but I suspect it will be at least until end of year,


until this works a little bit.


And similarly, we trust that rendering


will be more and more common across all kinds of web


services.


So at that point, dynamic rendering


is probably less critical for modern sites.


However, the best practices that we talked about,


they'll continue to be important here as well.


How does that sound, Tom?


That sounds really great.


I think that covers everything.


And I hope everyone in the room has learned some new approaches


and tools that are useful for making


your modern JavaScript-powered websites work well


in Google Search.


If you have any questions, we'll be in the mobile web sandbox


area together with the Search Console team.


And alternatively, you can always reach out to us online


as well, be it through Twitter, our live office hours


hangouts, and in the Webmaster Help Forum as well.


So thanks, everyone, for your time.


Which SEO tools and tests should every developer know?

Why can't SEOs describe the task that they want me to do clearly?


Why don't developers understand SEO basics?


Why do SEOs give us solutions that do not work with our tech stack?


Why do I have to wait so long for my development changes?


With me today is Crystal Carter, head of SEO communications at Wix.


Formerly working at Optics Solutions as a senior digital strategist


with years of SEO experience under her belt.


A Californian in the UK, she enjoys walks in the woods,


wildlife gardening, and perfecting her guacamole recipe.


Martin Shplitt is a developer advocate from Google's Search Relations team.


He's a geek who loves tech, old and new.


Living in Switzerland, he also loves cheese.


Hello and welcome everybody. With me today is Crystal.


I'm super excited to have you here.


I watched a talk of yours a while back,


and that really, really spoke to me as a developer.


I would love to talk to you about something that I observed


coming from the developer's perspective.


Oftentimes, SEOs barge into the development team,


and they're like, oh my god, we need to figure out our canonicals.


There's this other problem here with duplicate content.


We're going to get a penalty, blah, blah, blah.


Oftentimes, they are met with a blank gaze from the people they just spoke to.


What's happening there?


Why are we... We are working on the same thing.


We are all working on the same projects together,


and yet we seem to be so very, very different in terms of the vocabulary we use.


I absolutely agree. This is something that I've seen a number of times.


I think that one of the things that's tricky is that


we're sort of coming at it from different perspectives.


Sometimes if you think about it, it's a bit like a band.


If you have a band, you have your bass player,


you have your guitar player, you have your drummer,


and you're all trying to make a beautiful song.


But the bass player has different things that they need to do.


The drummer has different things that they need to do.


The drummer has different things that they need to do, and sometimes they overlap.


Sometimes they intersect.


But really, what we all need to focus on is making sure that we're making beautiful music together.


I think that it can be tricky.


I think sometimes it's because people who've built the website,


the developers behind the website,


very often they spend a lot of time and effort putting that together.


So it's quite tricky when someone comes in and says,


oh, it's broken, oh, it's wrong, it doesn't work.


And I think that it's really important to think about that


when you're speaking to developers as an SEO.


And one thing that I've found that's really useful is to think about it as opportunities.


So rather than saying this is broken or this is wrong,


say we have an opportunity to make the website faster if we do X, Y, or Z.


We have the opportunity to make the website easier for users to access in a different way


if we do this, that, or the other.


And I find that, generally speaking, if you go to it from that perspective,


then it can make those conversations a bit less brutal.


Yeah.


I love that. I love framing it as opportunities and possibilities to make things better.


Because fundamentally, that's what developers want to do in the first place anyway.


So that's really, really cool.


I like it better than framing it as a problem or a thing that is broken or needs fixing.


Because to be honest, it's also that developers are not even necessarily aware of these things.


It seems to be like you are preaching from the holy book that is written in a language that isn't ours,


and it comes out of nowhere and out of the blue.


One day, you're just implementing this feature,


and you're making sure it works in all browsers, works in all devices, is fast.


And then out of the blue, someone is like, it's broken.


So I like changing that into here's an opportunity to make it even better.


I do read up constantly on new technologies, on new things that browsers can do,


on new trends in design.


But I don't really even know where to get started on SEO.


And I think one of the things that us developers need to do more


is get out of our little bubble of development


and actually look at SEO as, at least at the technical side of SEO,


as something that is just part of what we deal with.


Not necessarily, it's not part of our job to do this, but we should know about it, I think.


Yeah, I think so.


And I think Google's really good at trying to bridge the gap between those spaces


and trying to make it more clear for people to understand how that works.


I think one of the things that's really useful


when you're thinking about developing an SEO


and understanding SEO as a developer is remembering that with SEO,


we're essentially taking the technology that a developer has built,


and we're interacting with it in the real world,


and users are using it in lots of different ways


and the ways that users are using the site,


the ways that users are interacting with the site


are going to be very different from, you know,


there'll be millions, billions of possibilities


for ways that people can access the site,


ways that people will search for the site,


ways for people that people will use the site


from what we're able to test in a sort of test development framework.


And so essentially, when, as soon as we build sites at my team,


and as soon as we build a site, they always say,


we built the site, can somebody run it through a few tests?


And I always run it through a few crawls,


and there's always a few things that we pick up


because the SEO is essentially thinking about,


you know, it's search engine optimization,


which means that you're thinking about how people search for


and arrive at the site


and how people interact with the information on the site


and how people use the site.


So SEOs will have lots of user data based on that site itself,


so let's say you're rebuilding a new site,


so we'll have lots of historic user data about how people use the site,


and then there'll also be general sort of industry user data,


so it might be that, you know, you've built a lovely shopping framework,


but we also know that within the shopping, you know, the e-commerce space,


that users have an expectation of being able to see this


and, you know, be able to pay in this way


and being able to sort of have a cart that's, you know,


really clickable and that sort of thing.


So I think that, you know, development's a really tricky job


because there's a million different ways that you could build a website.


Like, there's a million different ways that you can build an app,


the possibilities are endless, so, you know,


it's tricky to expect one developer or even one developer team


to know every single possible way you could build a website,


but when people are using the site,


that's where SEO and development interact,


and I think that developers can speak to SEOs


and ask them for data about, you know,


what are the sort of customer expectations for this type of website


or what are the, you know, how fast should it be?


So, for instance, e-commerce sites tend to be a little bit slower


than other websites, and that's because they've got


lots of different tracking things on them


and they've got lots of different out-of-work alternatives they're making.


We try, we try, but it's true.


So, you know, it's all sort of, it's all relative,


we want to make sure that we're meeting user expectations.


So I think developers can speak to SEOs and ask for more user data


to help them build ahead of time,


and I think that using some of the, like, Google has a lot of tools


that overlap, so like Chrome development tools I use all the time


when I'm speaking to developers, and, you know,


just things like View Source can be really, really useful in Chrome,


and that sort of helps bridge the gap between the two.


And I think that where there's the willing to understand


the two different parts of, you know,


both how users are interacting with it


and how businesses need a website to work


and how it's built in the framework and that sort of thing,


that, you know, there's opportunities for growth


and opportunities to make the web better for everyone.


Yeah, and you said two things that I want to address with you now.


Let's start with, so the one thing that you said is you run a bunch of tests


and there's a bunch of tools that developers can use,


and then the other thing is, like, there's a bunch of different solutions,


and I think those are two very interesting aspects.


Let's start with the tests, because you said, like,


once a website is done, I run a bunch of tests.


That's what we as developers do while we are building it.


So we have a bunch of tools that we are running


to, like, figure out bits and pieces, like how fast is this website,


where are problems coming from,


is it behaving the way that we expect it to behave,


and we actually write tests.


We write test cases such as if I click on this button,


and then I, well, if I click on this button, I should see this thing.


If I type something into this thing and then I click this button,


something should happen.


So, like, we are having a bunch of tests.


I wouldn't even know what tools or tests to use for SEO,


because for pretty much most of the things, I have stuff, right?


I can run a browser, basically remote controlled by code,


to, like, click on buttons and fill in form fields and then see what happens.


But what would I be looking for for SEO?


I actually don't know, and I don't know where to find out


or what tools are available to me.


I know Lighthouse has an SEO audit, but that seems very, very basic.


So what would you say is something that developers should look into


to support their SEO teams?


What are technical aspects that developers should be aware of?


Or where would they find out how?


Or what would they have to...


I don't even know what to ask an SEO.


So I would say that normally what happens when we launch a site


is I'll crawl it with something like Screaming Frog,


which will give you information about crawl errors.


And then the other thing is things like Search Console.


Search Console gives you lots of information about the way that Google is seeing it,


because different search engines will have different ways that they read websites.


Different bots will have different ways that they read websites.


So Search Console gives you lots of information about the way Google sees it,


and it can also give you information about how different users are seeing it.


So a page experience report, for instance, is based on real user information.


So it might be that within your test framework that everything looks perfectly fine,


everything's lovely and things.


But it's sometimes like if you've ever written something


and then you try to proofread it yourself,


sometimes you don't see your own writing things


because you've spent so much time writing.


Whereas if somebody else reads it, they'll go,


well, you've missed a full stop, or you haven't said and there or that sort of thing,


because they've got sort of fresh eyes.


And the other thing is that as a developer,


you're looking at it from a developer's perspective,


and you're fairly tech savvy,


whereas users are going to be coming at it from lots of different perspectives.


So I think it's really important to look at it


the way that it's rendering on different browsers,


the way that it's rendering on different machines,


the way that it's rendering on different mobiles and devices as well.


And yeah, tools like Search Console give you user data,


how people are using it.


And once a new website is live,


I think it's really important to think about how to expect to make further changes


and potentially expect to make further changes even six months down the line,


because you'll have more data based on it.


I think data is really important, particularly for businesses that are seasonal


or businesses that have sort of peaks and troughs.


So the way I think about that sometimes is like,


if you were to do data on a tree, like on an oak tree, for instance,


and you took all your data from between May and June,


all of your data would tell you that trees are green.


Oak trees are green, they have green leaves and they're great.


But if you do it for the whole year,


then you learn that sometimes they fall off and sometimes there's none.


No, sometimes the leaves aren't green at all.


So the more data you have, the better you can prepare.


And if you had the full data on the full year,


then you'd know that we'd also need to buy a rake to go with the tree.


I love that image.


That beautiful picture you're painting.


I do love painting pictures of trees.


Happy little trees.


Yes, I see.


We've gone into Bob Ross here.


Yeah, exactly. He's a legend.


That is awesome.


I remember it's really nice that you mentioned Google Search Console,


because I remember as a developer, again, coming from a developer background,


the very first time I came into contact with it


was when it was still called Webmaster Tools, if I remember correctly.


And I was at the same... This is going to sound really weird.


At the same time, I was overwhelmed and underwhelmed.


And it was bizarre, because I didn't even know where to look.


But I think that's exactly one of these interfacing moments


between SEOs and developers.


Because if you come to this tool and you see performance, impressions, clicks,


what's happening, then it might be a lot to take in at first.


But I think this is exactly where SEOs can guide the developers


or even just provide the data that they need for a specific conversation


to underline that specific conversation.


And I think this is what needs to happen.


And if it happens... I remember working with one specific SEO a long time back,


and she guided me through a bunch of the decision-making processes


that I didn't even know existed.


And that was very, very helpful.


And I think data collection, as you say,


if you look at the trees only in the summer months,


you might be very, very scared to see all the leaves fall off,


because trees are supposed to have leaves all the time.


I think the overwhelmed and underwhelmed thing


is something that I've encountered sometimes.


So sometimes when SEOs go to a dev, they say,


right, we want to make all of the title tags do this and this and that and that.


And they go, well, that only took me a minute to implement.


Is that really important? Does that really matter?


And it's like, from an SEO perspective, it's like, yeah, that makes a big deal.


Or that can make a big difference.


I've spoken to developers and I need you to add this particular bit of schema markup.


And they're like, well, I just copied and pasted it.


And I'm like, well, that's fine. That's great.


I'm glad that it didn't take you long to implement,


but it's also got a big impact on how Google crawls it and things.


So something that's super simple that a lot of developers miss


and something that I always check every time we launch a new website


is the sitemap.


And just literally putting a sitemap on the website.


Sometimes I explain it to people.


Like, if I went to a restaurant and I said, what's on the menu?


If they give me a menu, then I can figure out what's on the menu.


Or if I don't have a sitemap, then it's like just going to the restaurant


and them just being like, yeah, you know, have a look around


and you just see what other people are eating,


which is essentially what you're asking Google to do.


You're asking them to guess whether or not there's a Caesar salad on the menu.


And they'll probably figure it out.


Google's very sophisticated and everything.


But it's better if you give them a menu.


I like that, yeah. Again, beautiful analogy. That's lovely.


But going back to the testing and the tooling,


you mentioned a few tools and you mentioned Search Console.


Search Console, the bigger challenge is that I, as a developer,


can only use it once the site has gone live and Google has had a look at it.


Can I use things like crawlers? Which one did you mention?


Screaming Frog.


And I know there's like A.A. traps and all the others.


Can I use those in development as well, like before the site goes public?


Yeah, so in a staging setting, you can use Screaming Frog to sort of understand


if you've got gaps with things like with page structure things.


So for instance, H1s are super important.


I think people very often underestimate that.


That's another one that I very often see.


I very often don't put an H1 on the homepage.


And I'm like, that is an easy win. Please put an H1 on your homepage.


And so, yeah, you can use Screaming Frog to see a lot of different things.


There are many, many, many, many, many different layers


and many different configurations of your Screaming Frog setup.


So you can see a lot of the structural things like headings and title tags


and meta descriptions and, you know, URL structure and that sort of thing.


And you can see all of those sorts of things.


And then you can also see the navigation and various other elements.


And Screaming Frog is really useful with helping with the sort of understanding


how the like the crawl depth through different pages.


And so this goes back to indexation, which is about your website in the real world.


So again, another metaphor. I speak in a lot of metaphors.


But sometimes I think of it like the developer builds a plane and the pilot flies it.


And so like the developer, like there, you know, the SEO sort of flies it.


And like in the real world, like the way that it interacts with, you know,


various different things is different.


So, yeah, that's something that's really useful for understanding how the site is performing


and how it's crawlable and how it's readable online and before you go live.


But in a migration setting, it's also useful.


There's a couple of different Chrome extensions that are really useful


for sort of understanding like what the page was before


and like what the page is now and versus what it's going to be in the future.


So I was working with a client the other day and they were talking about changing their homepage,


which isn't a full migration, but changing a homepage can change a lot


because on most websites, the homepage is the most visited page.


So if you change the number of links on that page, then that can change, you know, the crawl depth.


And that can change the performance overall.


And so, yeah, there's a couple of different tools.


There's a tool that I use called SEO Pro Extension, which is really, really good.


Christina Azarenko put it together, I think you interviewed her previously on another thing.


Yes, it's fantastic. And yeah, so I use that a lot.


So that can tell you what you've got now. Also, a tool that I use particularly with migrations,


if you're thinking like, let's say you develop something and the client says it's not as good as it was before.


But that website is gone. Something called Wayback Machine is really good.


I've used that a few times to sort of understand why a site before was working and why it's not now.


So there was a site that we had where they used to have a sort of most recent content feed showing on the homepage.


And that helped with crawl depth and helped with sort of, you know,


retincy of new content and that sort of thing. And then when we moved to the new website, it didn't have that.


So when we moved to Wayback Machine, I was like, we need to put that back on.


So that that goes back. So that gives us more tools.


We always think that we need a super-whizzy technical fix.


And sometimes it's not necessarily the case that you need something super fancy.


You know, I know there's a lot of AI at the moment and there's a lot of automation and lots of things like that.


But sometimes you can do something pretty straightforward that will fix it.


So, for instance, you know, I've seen it before where, you know, from an SEO point of view,


there's like we're looking around and we're going, oh, there's a bunch of 404s and oh, we need to fix this.


The developer needs to sort something out or something. It's like, well, you know, if there's a 404,


like if you've got loads and loads of 404s, like let's say most of the pages on your website have like seven links


and there's four hundred or seventy five 404s to this one URL, it's probably in your menu.


It's probably in your menu. You've probably got a 404 in your menu.


You probably just need to change the link on the menu. And, you know, sometimes people think, oh, you know,


we need to like do this whole thing and fix that. It's like you might just need you might literally just need to change one little thing


that isn't even a dev thing, for instance.


So I think that that that, yeah, I think I think thinking about about what is really genuinely required from it,


from a developer point of view is really, really useful. And also, but I was also thinking about about, you know,


how complex the solution might be.


So, for instance, in the in the in the situation you were talking about where where, you know,


you built it one way and and there's and there's a and the SEO wants a bunch of other things.


If your if your tech stock is is is is resistant to the implementation that you want,


then there might be an external tool that you can use that solves the solution or, you know, that gets that gets you where you need to go.


Or there might be another way that you can that you can do it.


So, for instance, I had a I had a situation where, you know, I wanted an RSS.


We were trying to build an RSS, you know, directly on the website.


And we ended up using a third party tool and it works fine. It does exactly what I need it to do.


And and we'd, you know, or I could have spent a month trying to get someone to build one from scratch doing this.


But why? But why? I got a third party tool. It works perfectly fine.


It has and it doesn't cause, you know, there's no there's no third party.


You know, I don't have any third party lag from a JavaScript. It's literally just one little line and it's totally fine.


And, you know, I had another. Yeah.


And I find other times where, you know, like, for instance, structured data is something where, you know, everybody's going JSON, JSON, JSON.


And JSON is fantastic. But if the if the website already has a micro data set up, you might get you might be able to move the needle close, like quick, more quickly by just updating the micro data if it's already if it's already in place.


So, you know, I think we've got it's useful from an SEO point of view to think about not only what moves the needle, but also what moves the needle efficiently.


And from it, from from a tech point of view and and from and from an SEO point of view.


And I really like that statement because I wish a lot of the SEOs that I had to work with in the past would acknowledge that.


And oftentimes I feel like there is a bit of dogmatism in the way that some people operate as in like, as you say, you identify the thing that moves the needle and moves the needle efficiently, like what is what is easy to do, but like has lots of impact.


And we discussed that in a previous episode as well.


But the thing is, oftentimes, it seems that people SEO specifically go from half knowledge of someone on the internet said, or it used to be like that 10 years ago, and then just ask for a solution for something that isn't even a problem in the first place.


JavaScript is one topic that invites this kind of situations, it seems it's like, there's still a lot of people wandering around like, Whoa, no, we need to implement dynamic rendering or server side rendering for this client side rendered application.


Because unless we do that, it won't show up in search. And then I'm like, but if I Google for a thing on your page, then it shows up.


So it's already there. And I see that it has traffic in Google Search Console, it gets the impressions, it gets the clicks. Why? Why are we doing this? Oh, because Google doesn't understand clients like render JavaScript applications. And I'm like,


but the data clearly shows that it does. Right? It's there. I'm not saying that it's true for all the situations and all the cases and all the pages. Yes, for some pages, it still matters for some, it still makes a lot of sense. If you see a problem, then this might be a solution to that problem.


If the client side rendering really is the issue that you're dealing with. But oftentimes, it just isn't.


You spend a lot of time, a lot of effort, a lot of money on implementing something that invites more trouble for solving nothing.


Why does that happen? I think sometimes people will hear a buzzword or they'll hear like, oh, yeah, you know, we really need to sort out this and that.


And so I think sometimes people get excited about a new thing. And, you know, I mean, I can hold my hand up. There's definitely been a time when I've done that.


But the other thing I think is would be really, really useful, which I would appreciate as an SEO speaking with devs, is that if you don't know what I'm talking about, tell me that you don't know what I'm talking about.


Right. Yes. A hundred percent yes. Yes, please.


And also if you need me to give it to you in a different way, please let me know. There's a lot of tools that will help you to do that. So I use a lot of screen recording tools.


So like Loom is one that I use a lot. And I think Slack now has an integration where you can just record your screen really, really quickly and send it in a Slack message.


And that makes a really big difference. Chrome developer tools is fantastic. I do tons of PowerPoints where I'm just like, this is here and that is like this and this is like that and that is there.


And sometimes if you can see it and if you can show them the code, then that then even if you don't know the exact words, they can figure out what you're talking about.


But it's also useful to learn some of the words as an SEO. So, you know, like learning what is and isn't a variable. Learning about what's going on with your server, what's going on with your, you know, like how is it deployed?


Like how is all of that working? It really helps from an SEO point of view for you to get things actioned quicker if you can give the developers more information.


So, for instance, there's a couple of sites where, you know, I have access to the folders and things. And I can see I can root around in Bitbucket or whatever.


And I don't change things very often. But I can root around. But for some, like I have some clients, I've had clients where they still had a sort of static site map.


And I was generating the static site map. There's smaller sites or whatever. And, you know, if that's something that's taking dev resource, you can ask them, like, can I have access, please?


And then you can learn how to do it. And you can do it. Like creating a static site map is a really simple operation. Uploading it is really simple.


But, you know, waiting around for one can be quite a pain. So, if you can do it as an SEO, then that's really, really great.


So, you know, there's some things where you can ask for access to certain parts of the stack so that you can give them better information.


But again, that's about building trust. So, if you have the conversations with your developers, you know, before things get heated, before things are broken,


before you start moving around, like everything, everything, then it really helps.


And I mean, even if you're working with remote teams, like we work with a few, I work with a few remote developers, we have developers in house, but we also have developers that we work with for, you know, that I work with for other clients or developers that work in other countries.


And we use Slack and email and stuff. And I will literally send like, you know, emojis and like, hooray gifts and like, and that sort of thing.


Like if they fix something, I'm like, oh, all right, thank you so much. You know, like say thank you. Thank you gets you so far.


Same for developers towards SEOs, by the way, if I made it a point whenever I was like, something was pointed out to me where we could do better than we just did.


And then we saw the business impact. And then it's like, thank you very much for bringing this to our attention. And thank you a lot for guiding me through the process.


Because oftentimes, I didn't know why I was doing something or how exactly it's expected to be and where I would verify if it's done correctly.


And then having someone to guide me through that was great. And I was grateful for it. So say thank you.


Right, absolutely. And if you work really well, then like as a developer, like that's, you know, that's adding another another string to your bow.


Like, then it's something that, you know, it just keeps it makes everything it makes everything better.


So, so yeah, I think that I think that that building it's really important to build good relationships in that way.


Absolutely. So I think it's important to, to, to a communicate if you don't understand something, be it you as an SEO or you as a developer, if you're not sure about something, just ask, find the tools that the other people are using.


Just ask your developers, what tools are you using? Ask the SEOs, what tools should I be using to check when I'm when I'm building this or implementing this or fixing this? Phrasing it as opportunities of improvement is probably much, much better than phrasing it as a problem or a challenge.


And yeah, if we just work together and finding the right solutions together, I think we are doing each other big, big favors. I guess that's, that sums it up quite nicely.


Absolutely. Absolutely.


Thank you so much for joining me for this conversation, Crystal. This has been really, really exciting and interesting. And I do hope that everyone out there got something out of it as well. I certainly did.


And I am looking forward to see what other cool things you will be sharing with the community and us in the future. And yeah, again, thanks a lot for joining and take care. Have a great time. Bye bye.


Thank you so much, Preston.


I was just gonna say I'm a huge fan.


The Future of SEO

Welcome, everyone, to the next episode of the Search Off


The Future of SEO

the Record podcast.


The Future of SEO

Our plan is to talk a bit about what's


The Future of SEO

happening at Google Search, how things work behind the scenes,


The Future of SEO

and who knows, maybe have some fun along the way.


The Future of SEO

My name is John Mueller.


The Future of SEO

I am a search advocate on the Search Relations team


The Future of SEO

here at Google in Switzerland.


The Future of SEO

I'm joined today by Martin and Gary, both also on the Search


The Future of SEO

Relations team.


The Future of SEO

And today, we'll be talking about the future of SEO.


The Future of SEO


The Future of SEO

The big changes I've been seeing over, I don't know,


The Future of SEO

maybe the last 10 or so years is how more and more sites are


The Future of SEO

moving to hosted platforms where basically you


The Future of SEO

don't have to run your own server anymore


The Future of SEO

or where it's almost better that you don't run your own server


The Future of SEO

anymore, because you don't have to deal with all


The Future of SEO

of the technical infrastructure that's


The Future of SEO

associated with everything around a server.


The Future of SEO

And to me, that seems like, I don't know,


The Future of SEO

It seems like a reasonable move because you're


The Future of SEO

kind of offloading a lot of the technical details,


The Future of SEO

and it gives you more room to focus on what actually matters,


The Future of SEO

like the content that you're creating.


The Future of SEO

So I guess in a sense, if we think about SEOs,


The Future of SEO

it means SEOs won't need to learn HTML anymore, right?


The Future of SEO

Oh, no, no.


The Future of SEO

No.


The Future of SEO

Abort mission, abandon ship.


The Future of SEO

No, John, no.


The Future of SEO

Well, I mean, it's like if you just have a rich editor


The Future of SEO

and you just type things in, and then you format your text


The Future of SEO

properly, and you add some links,


The Future of SEO

what do you need to do with HTML?


The Future of SEO

I mean, SEO is more than just doing the text part, right?


The Future of SEO

It's not just about writing the content.


The Future of SEO

Basically, if you hire a copywriter,


The Future of SEO

that could do the content for you.


The Future of SEO

But SEO is also about link tags and meta tags and title elements


The Future of SEO

and all those weird things in the head section of the HTML


The Future of SEO

that you can put there.


The Future of SEO

So you kind of want to know about them


The Future of SEO

to control how your snippets look like


The Future of SEO

or how your titles show up in search results


The Future of SEO

and the rel canonical tag to control what will be the,


The Future of SEO

or what should be the canonical version of a URL.


The Future of SEO

You kind of want to know that.


The Future of SEO

But couldn't you just have that in your CMS?


The Future of SEO

It's like you just have a big field for text,


The Future of SEO

and then you have some extra fields for metadata.


The Future of SEO

But if you just started a website,


The Future of SEO

then do you want to learn, for example, what is rel canonical?


The Future of SEO

Could you explain to me as an absolute noob


The Future of SEO

what is a canonical URL and what is its relation


The Future of SEO

to rel canonical?


The Future of SEO

It's very hard to simplify it.


The Future of SEO

Or hreflang, for example, try to explain hreflang to someone who


The Future of SEO

just came to the internet and they've never even thought


The Future of SEO

about localization before.


The Future of SEO

I mean, explaining hreflang to someone


The Future of SEO

who has been on the internet for a decade is hard too.


The Future of SEO

I mean, there are people who understand it and can


The Future of SEO

do a good job.


The Future of SEO

That's true.


The Future of SEO

Also, having worked with CMSs in the past,


The Future of SEO

I've seen that the ones that are pretty much the more successful


The Future of SEO

ones allow you to always insert custom HTML,


The Future of SEO

because an editor that needs to be as simple as possible


The Future of SEO

for everyone to use it and to create content


The Future of SEO

should also give you the opportunity to say,


The Future of SEO

and now I want to jump out of the simple mode


The Future of SEO

and actually go and do something more advanced.


The Future of SEO

And that usually contains HTML or means


The Future of SEO

that you need to write HTML.


The Future of SEO

So if you don't know any HTML, you'll


The Future of SEO

be very, very quickly out of your depth there.


The Future of SEO

So I think that's a risky direction to take.


The Future of SEO

Also, when we are launching new meta text, for example,


The Future of SEO

or directives, robots directives, for example,


The Future of SEO

then we kind of rely on SEOs to make use of them instead


The Future of SEO

of site owners use them.


The Future of SEO

And we kind of tailor the documentation


The Future of SEO

towards SEOs versus site owners.


The Future of SEO

Like, for site owners, if you look on Dev site in our search


The Future of SEO

documentation, then the site owner tailored content


The Future of SEO

is very simplistic and for a good reason.


The Future of SEO

They need to get the basic information that


The Future of SEO

can get their site do OK in search.


The Future of SEO

But then if you want to take the next step,


The Future of SEO

then you either learn more about search


The Future of SEO

and how you can do stuff on your website that makes


The Future of SEO

your site do better in search, or you hire an SEO.


The Future of SEO

And then we have the SEO documentation,


The Future of SEO

which is much more in-depth, I guess,


The Future of SEO

and explains things differently with the assumption


The Future of SEO

that the person who's reading it already knows stuff about search


The Future of SEO

and how things work and how things connect to it.


The Future of SEO

You probably don't want to throw the site owner into that water


The Future of SEO

just when they started with their site.


The Future of SEO

OK, so I guess taking a step back,


The Future of SEO

you're also saying that SEOs should know


The Future of SEO

HTML from the beginning, even now.


The Future of SEO

Yeah, I would say so.


The Future of SEO

I think it's one of the fundamental technologies that


The Future of SEO

make the web what it is.


The Future of SEO

And if you want to work with it, you


The Future of SEO

should know at least a little bit.


The Future of SEO

Yeah.


The Future of SEO

It's not like JavaScript, where you


The Future of SEO

don't have to know JavaScript to create a good website.


The Future of SEO

I love Martin's face.


The Future of SEO

But with HTML, you can't avoid HTML.


The Future of SEO

It's the very basic building block of the web.


The Future of SEO

So HTML is not going to go away?


The Future of SEO

No.


The Future of SEO

It's also such a progressive technology in the sense


The Future of SEO

that with a little bit of it, you


The Future of SEO

can do a bunch of stuff already.


The Future of SEO

And then you can expand step by step.


The Future of SEO

Whereas, for instance, JavaScript, you just


The Future of SEO

need to know a lot of stuff to actually get something going.


The Future of SEO

And if you make a mistake, potentially your entire code


The Future of SEO

doesn't work.


The Future of SEO

Whereas in HTML, the browser will


The Future of SEO

do its best to actually still make it work somehow.


The Future of SEO

So the more you know, the better it is.


The Future of SEO

But I don't think everyone needs to be an HTML expert.


The Future of SEO

You just need to understand what HTML is and how to use it.


The Future of SEO

OK.


The Future of SEO

So John, do you think SEOs don't need to know HTML?


The Future of SEO

Oh, I don't know.


The Future of SEO

I mean, it's a question that comes up every now and then.


The Future of SEO

And I think a part of the reason behind that question


The Future of SEO

is also that there are just so many different things


The Future of SEO

that SEOs do.


The Future of SEO

And some of them focus purely on content


The Future of SEO

or focus purely on building relationships


The Future of SEO

with other websites.


The Future of SEO

And for some of those activities,


The Future of SEO

you don't need to know HTML.


The Future of SEO

But there are also lots of technical on-page things


The Future of SEO

that you do have to use.


The Future of SEO

I mean, I don't know how that will evolve,


The Future of SEO

if those directions will kind of merge more or separate out


The Future of SEO

more.


The Future of SEO

But yeah, I mean, it's always interesting to see.


The Future of SEO

HTML is not going to go away.


The Future of SEO

So you might as well get used to it.


The Future of SEO

And if as an SEO you don't know anything about HTML,


The Future of SEO

then maybe it's time to actually try things out.


The Future of SEO

Another thing that I think is happening more and more


The Future of SEO

is that I think the user is kind of expecting to be able to use


The Future of SEO

any app that they have in any platform,


The Future of SEO

any device that they use.


The Future of SEO

And it feels like that kind of work


The Future of SEO

is going to continue as well.


The Future of SEO

And probably that means things like understanding JavaScript


The Future of SEO

will become more and more important for SEOs as well,


The Future of SEO

which will probably make Martin happy.


The Future of SEO

But it probably also means that a lot of these apps


The Future of SEO

are going to be more and more important for SEO in general.


The Future of SEO

Like, what do they actually want to have findable on the web?


The Future of SEO

Because in the past, they were just apps.


The Future of SEO

MARTIN SPLITTENBERG Yeah, that's actually a big topic


The Future of SEO

and a very interesting one, and one that I think


The Future of SEO

has been in the past underrepresented


The Future of SEO

or underinvestigated or under-researched.


The Future of SEO

In the future, I think a lot of our applications


The Future of SEO

will just happen to run in the browser.


The Future of SEO

APIs and opportunities, you can have


The Future of SEO

a video chat in the browser.


The Future of SEO

No one has to necessarily install a client or a desktop


The Future of SEO

app or a mobile app to do a video chat.


The Future of SEO

Video chats have been quite popular


The Future of SEO

in the last couple of years.


The Future of SEO

And I think that has shown that applications


The Future of SEO

can shift to the web, even if they are a little more


The Future of SEO

intricate.


The Future of SEO

But then the question becomes, how


The Future of SEO

do you represent that to someone who searches the web?


The Future of SEO

And what kind of content do you want to highlight?


The Future of SEO

And we had this challenge at a company


The Future of SEO

that I worked for in the past, where they would create


The Future of SEO

interactive 3D models of real estate spaces,


The Future of SEO

like apartments or offices.


The Future of SEO

And you could furnish them and walk around them.


The Future of SEO

You could basically do a virtual viewing in the browser


The Future of SEO

of a different space.


The Future of SEO

You could even do that in AR and try out


The Future of SEO

different furniture in your own home,


The Future of SEO

and all of that in the browser.


The Future of SEO

And then the question became, OK,


The Future of SEO

so we have this amazing application that


The Future of SEO

happens to run in the browser.


The Future of SEO

But if a search engine looks at it, because it is all visual,


The Future of SEO

it is a black box for a bot for a computer.


The Future of SEO

As far as Googlebot or any other computer looking at the website


The Future of SEO

would go, they would see a title, a meta description,


The Future of SEO

and then a canvas, which is effectively


The Future of SEO

a big rectangle of pixels that they


The Future of SEO

have no idea what they represent or what they mean.


The Future of SEO

And then the question became, how do we get that


The Future of SEO

into a search engine?


The Future of SEO

So for instance, we made a virtual model


The Future of SEO

of Don Draper's apartment in Mad Men.


The Future of SEO

We also made the Simpsons family home a 3D model


The Future of SEO

that you could visit in the browser, which


The Future of SEO

is really, really cool.


The Future of SEO

But if you search for the Simpsons house or Don Draper's


The Future of SEO

apartment, you wouldn't really find our 3D model,


The Future of SEO

because as far as Google search and other search engines


The Future of SEO

are concerned, our page is really, really low on content


The Future of SEO

and not very relevant to the query that you enter.


The Future of SEO

So how do you do that?


The Future of SEO

And then there's obviously a bunch of strategies,


The Future of SEO

but I think that will be a topic in the future where


The Future of SEO

SEOs need to identify together with the people making the app


The Future of SEO

and using the app on what content do we want to expose,


The Future of SEO

how do we expose it so that it is useful and understandable


The Future of SEO

to search engines.


The Future of SEO

And that will also be something that search engines will work


The Future of SEO

on.


The Future of SEO

And we can already see that 3D models for products,


The Future of SEO

or like white sharks or tiger or whatever,


The Future of SEO

there are 3D models for this so that you


The Future of SEO

can get a spatial feeling for how these things look


The Future of SEO

and interact with their environment.


The Future of SEO

Maybe we'll see more of that in the future.


The Future of SEO

I don't know.


The Future of SEO

But it's going to be a huge field that


The Future of SEO

is really, really interesting.


The Future of SEO

And with that, also a lot of JavaScript


The Future of SEO

comes to the web that we as a search engine


The Future of SEO

need to understand and run.


The Future of SEO

And that might mean that technical SEO might


The Future of SEO

get a lot more technical and a lot more tricky in terms


The Future of SEO

of how they work with JavaScript.


The Future of SEO

But it sounds like something that not many people, I think,


The Future of SEO

have thought about at this point.


The Future of SEO

So it sounds a lot like you have to combine the SEO strategies


The Future of SEO

that you have for existing sites suddenly with completely


The Future of SEO

different site models where maybe


The Future of SEO

the developers have purely focused


The Future of SEO

on the technical aspects, like how does this 3D model actually


The Future of SEO

work?


The Future of SEO

And at some point, you combine it


The Future of SEO

with the marketing side of SEO and kind of like,


The Future of SEO

how do I package information in here


The Future of SEO

so that text-based search engines can actually


The Future of SEO

make use of that?


The Future of SEO

MARTIN SPLITTENBERG Yeah, and also, how can users


The Future of SEO

navigate this application?


The Future of SEO

Because if you give me, let's say,


The Future of SEO

like an empty paint application, then I


The Future of SEO

don't necessarily know if that is the application I need


The Future of SEO

to do what I need to do.


The Future of SEO

So then how do you package this content?


The Future of SEO

How do you interlink functionality with content?


The Future of SEO

And yeah, that's going to be interesting to see.


The Future of SEO

JOHN MUELLER


The Future of SEO

Cool.


The Future of SEO

OK.


The Future of SEO

So that's like one area where I guess SEOs will be required,


The Future of SEO

almost.


The Future of SEO

That sounds pretty good.


The Future of SEO

What kind of things do you think will stay the same?


The Future of SEO

So you mentioned HTML.


The Future of SEO

Both of you are like, the future will


The Future of SEO

be based on HTML on the web.


The Future of SEO

MARTIN SPLITTENBERG I think so.


The Future of SEO

JOHN MUELLER JavaScript you mentioned,


The Future of SEO

is JavaScript going to be the thing,


The Future of SEO

or is there going to be a new Flash kind of technology that


The Future of SEO

comes out and replaces JavaScript?


The Future of SEO

MARTIN SPLITTENBERG I think as JavaScript has evolved so much


The Future of SEO

over the last decades, I pretty much


The Future of SEO

think that it will continue to evolve.


The Future of SEO

And efforts at replacing it, like Flash, or Java,


The Future of SEO

or Moonlight, Silverlight, and all the other lovely things


The Future of SEO

that happened have historically been not very successful.


The Future of SEO

So I would be very surprised to see that go away and be changed


The Future of SEO

for something else.


The Future of SEO

OK.