3 Quick Ways to Create a Sitemap – Chalk Talk Thursday #9
On today’s Chalk Talk Thursday, I wanted to cover a pretty basic SEO task, which is how to generate a site map. In the video above I run through a few common ways to go about this.
There are a bunch of ways to generate sitemaps, and I’m going to introduce you to several. I’m going to cover how to do so with a WordPress plugin because it’s very, very common. I’m going to talk about how you can use Yoast and how to properly calibrate it and all that good stuff. I’m going to talk about a free tool, which is actually quite good and very, very useful for smaller sites. I’m also going to talk about how to use Screaming Frog, which is what I use the majority of times, especially when I’m dealing with a bigger ECOM site. Let’s dive in.
1. A Quick and Easy way to Generate a Small Sitemap Using XML-sitemaps.com
The first tool I will mention is xmlsitemaps.com. This is really handy, and actually works perfectly fine for most small sites. It’s a free tool. You just hop over to xml-sitemaps.com, plug in the URL that you actually want to crawl through, and then it’ll do a little crawl for you. For smaller sites, it’s free. Of course, they have pro plans, so check those out if that’s of interest to you, but if you just have a small site and you really quickly want to get a site map made, that’s the way I’d proceed. If you’re dealing with Shopify or BigCommerce and most other tools out there, they’re actually going to have some kind of functionality that’ll generate a site map for you so you won’t have to worry about this.
2. How to Build Your WordPress Sitemap with Yoast SEO
I like to do something like this though just because it can often be useful to get a custom site map, especially when some installations are creating bloat. I’ll talk about that a little more with WordPress. Inside of WordPress, there’s a really, really handy plugin called Yoast SEO very commonly used, very, very helpful as far as a few different services go. If you want to get title tags and meta descriptions on a site, it can be handy to use Yoast SEO. There’s a lot of features here. They obviously want to sell you some stuff, but for the free plugin, you can actually get a lot of functionality out of it, and it can make implementation of SEO items extremely easy.
Click on General, and you’ll see this screen, and there’ll be a few different settings connected here, which you can check out. After fiddling with the settings, click on the little question mark to proceed.
After clicking on the question mark, they’ll automatically generate an xml site map for you. This is what you want to plug in to search console. The one thing that Yoast does that’s actually really nice is split up your pages, so inside of search console, there’s the index they’re giving you, and then they’re breaking that into the multiple page types, so you have post, and then you have pages, if you have custom post types or events, all of that kind of stuff. They do a nice little job of chunking your site map, and then inside of search console, you can see what your index ratio is of each specific page type, which is really nice.
Through Yoast, it’s easy to get a sense of how many of your pages are properly indexed. If you find that you only have 60 out of a hundred of your posts indexed, you might say to yourself that there’s probably some kind of issue here.
Yoast makes it easy to filter your sitemap and find the pages you most want indexed. The other thing that you should keep in mind is Search Appearance and the secondary tab labeled Content Types. Now, one thing that’s very helpful about Yoast, especially when you’re just hopping into the SEO world for the first time, is they make it very easy to auto-generate title tags and meta descriptions. They have these nice little cues, and then you can also go into your page and your posts and customize there as well, which is even better. But the key thing that’s really cool is you can go into here, Yoast features nice little cues that point you in the correct direction; and the service makes it easy to deactivate things if they’re unimportant so you can keep them out of the index.
If you say, “No, I don’t want to see testimonials,” that’ll basically block it out of the site map that they generate, so all the bulk can be quickly thrown out of the site map, leaving just the gold on Google’s side. That also lets you know, “If I don’t have these 50 pages that really, really matter to my index, but I just have 30 of those,” then that’s a problem, rather than when you have a bunch of bloat that’s going to hurt the small chunk of pages you really care about. In that situation, you’re going to have really imprecise data as to what actually is getting indexed, and then you’re not going to know what to optimize for.
This is probably the most common way that I see site maps being implemented on a WordPress installation. Shopify and BigCommerce will have different systems, but they do have some systems in there that you can use, and then if you just have a smaller site, this is a really useful tool, xmlsitemaps.com. You can just download it, and it’s ready to roll. Very, very nice functionality.
The other cool thing about Yoast is the update system. It’s quite easy to reping the system through the console. And as you’re retesting the site map from the search console side, you’ll be able to see these pages getting updated, and you don’t have to do a lot of work.
3. For Large Businesses Scale up Your SEO Needs with Screaming Frog
The final way of building a sitemap that I’ll cover is using Screaming Frog. Far and away, this is my favorite way to create a site map. Screaming Frog is really good on a super large ECOM store. Even when developers come to me, and they’re like, “Shaheen, I already made a site map,” or, “It’s auto-generating and renewing all the time,” blah, blah, blah… I typically go straight into Screaming Frog, and I make a site map from here.
Now, I’m just going to plug my site in here, which is pretty new and there’s not a lot going on there, but the really useful thing about screaming frog is you can crawl through the site. Once you have your total list of pages on the site, can do an export, and then you can comb out and delete URLs that you don’t want crawled because you think they’re useless or there’s just bloat. It lets you be really, really refined about which URLs and which images you want to include in your site map.
As far as pruning your site map for performance goes, this is very helpful and also exactly what you want to be doing on a larger ECOM store. You may have products that you need to be indexed, but may have subcategory pages that you need to ignore. Or you may have a ton of blog posts that are excellent linking out to some product pages, and you want to make sure that you have everything covered; but then you also have some weird archive folders and some legacy news content that your SEO wants you to keep on the site but you don’t really need because no one cares about it. So you can throw that stuff out and only crawl the URLs that really, really matter so that you’re actually getting what’s important inside of the search engine.
I’m just going to stop this real quick so I can show you what this pruning looks like. Basically, you go over here, and just say site map crawl, do a little export. We get our list of URLs. Open that up. Then we’re going to have a list of URLs that we want to work off of. Then you’re free to select and delete the one’s you don’t care about.
Let’s say these are the only pages that I really care about indexing on my site, which is obviously not true because there’s so much more going on there, but we’ll just take a little chunk here, and let’s say that I have like 80 pages, but these are the only ones I really care about. I can copy that list of URLs, go back up to Screaming Frog, click on Mode, and then go to List, and I can paste in all the URLs that I just copied over. It’ll dump those into the system. You go to Upload, and then go to Paste. It’ll say 27 found. We might have 100,000 or something that we really care about out of 230 if you’re working on a big ECOM site, whatever it may be. You go through there, and then it’s only going to crawl that list of URLs.
Important Note: Screaming Frog has both a free version and a paid version. You can’t use the free license to do crawls, but the service can still generate a site map that can cover up to 500 URLs, which is more than enough for most people.
Anyways, we crawl through our URL list that we really care about. Let’s say it’s, could be 10,000, could be 5,000. Then you’re going to hop over to the very, very top of the screen right over here. Click on Site Maps, and the click on XML Site Map. This is the instruction and configuration for making a site map. Again, the system inside of Screaming Frog is just awesome. It deselects 300s and 400s and 500s by default. This can also be a really good test. If you use the Screaming Frog tool to crawl a site map and you see a bunch of 400s, that lets you know that you’re coming into a situation where the dev probably isn’t very good at generating a site map.
I had a situation recently where the devs insisting that they had a site map made and that it was great. Went to the site, and found that basically the site map was only suggesting that Google index a very small percentage of all the products and categories on the site. It wasn’t a thing where pruning had been done. It was just negligence on their part.i
Anyways, you can specify if you want to include PDFs in your site map, if that’s a very important part of the experience.
There’s also priority. This is a really nice feature of Screaming Frog. Definitely more important on big sites, but as the crawl depth goes down, so you have, say, the home page, category pages, subcategory pages, sub-subcategory, and then product and services, you’re able to set the priority
Adjusting priority can be very helpful on bigger sites where you’re dealing with how to actually manage the “crawl budget,” that’s coming in from Google, and really, it’s just about making sure the right pages that are getting frequently updated that you really want to rank are getting indexed properly by Google. That’s all that’s really happening there. It’s not like you’re going to completely destroy yourself with a bad site map, although you can cause some issues, but it’s not going to be an instant death. There’s other SEO things that are in place that should keep you from having any big failures if your site map isn’t perfect.
You can also check to include images. This is actually a functionality I don’t like as much in Screaming Frog. Keep in mind that they set this up to a certain number of inlinks, so this can be useful if you select to include relevant images with up to two inlinks. The dangerous part is logos. Let’s say credit card pictures for your payment portals or the cart button will get indexed as images because they have tons of inlinks in them, so just be careful with this. And then hreflangs can be very important on bigger international sites. Ultimately, though, in site map creation, you really want to go with Screaming Frog because it gives you all the functionality as far as crawling and pruning goes.
Pro Feature: Be sure to try out the list mode in Screaming Frog, there’s actually an option to download a site map index. This can be really helpful, again, just as far as searching out and debugging any kind of mismatch. You just give it a URL, and then it’ll download all the URLs based off of the site map that you’re pointing it, and it’ll crawl both the site map index and the site map.
Anyways, I hope that was helpful. I know that’s super basic, but just good coverage, something that everyone should know. Again, great free tool on xmlsitemaps.com. If you have a WordPress site, I would absolutely just go with Yoast and go with their functionality as far as site map generation goes. Just make sure that you’re pruning what’s actually in the site map on the search appearance, content pages, and just make sure you’re DNXing the stuff that really doesn’t matter, that bloat that WordPress is creating. Then if you have a bigger site and a bigger need, absolutely go with Screaming Frog. Still worth a shot if you have a site you just want to check for errors, but this is really the best tool to go with if you have a massive site and you really want to prune and focus and you want to get tricky with how you’re having Google index your site. I hope that was helpful.