How to define All Existing and Archived URLs on a Website

How to define All Existing and Archived URLs on a Website

Blog Article

There are plenty of motives you could will need to uncover all of the URLs on a website, but your exact objective will decide That which you’re hunting for. For illustration, you might want to:

Recognize each indexed URL to investigate concerns like cannibalization or index bloat
Collect current and historic URLs Google has observed, especially for internet site migrations
Obtain all 404 URLs to recover from post-migration errors
In Every scenario, one Resource won’t Provide you with every thing you need. Sad to say, Google Search Console isn’t exhaustive, plus a “site:instance.com” lookup is limited and tough to extract knowledge from.

During this write-up, I’ll wander you through some resources to build your URL record and in advance of deduplicating the information employing a spreadsheet or Jupyter Notebook, depending on your internet site’s dimension.

Aged sitemaps and crawl exports
In case you’re trying to find URLs that disappeared with the live internet site lately, there’s a chance somebody in your group could have saved a sitemap file or possibly a crawl export prior to the alterations had been made. If you haven’t now, check for these information; they will frequently deliver what you may need. But, in the event you’re looking through this, you most likely did not get so lucky.

Archive.org
Archive.org
Archive.org is an invaluable Instrument for SEO tasks, funded by donations. If you search for a website and choose the “URLs” selection, it is possible to obtain around 10,000 listed URLs.

On the other hand, there are a few constraints:

URL Restrict: You can only retrieve around web designer kuala lumpur 10,000 URLs, which can be insufficient for bigger websites.
High quality: Numerous URLs might be malformed or reference useful resource information (e.g., images or scripts).
No export possibility: There isn’t a developed-in solution to export the list.
To bypass The shortage of an export button, use a browser scraping plugin like Dataminer.io. Even so, these restrictions indicate Archive.org may not offer an entire Option for much larger web-sites. Also, Archive.org doesn’t point out no matter if Google indexed a URL—but when Archive.org located it, there’s a good possibility Google did, much too.

Moz Professional
While you may perhaps normally utilize a link index to seek out exterior web-sites linking to you personally, these tools also learn URLs on your internet site in the method.


Tips on how to utilize it:
Export your inbound inbound links in Moz Professional to acquire a swift and simple list of goal URLs from the web page. For those who’re working with a huge Site, consider using the Moz API to export facts outside of what’s manageable in Excel or Google Sheets.

It’s crucial to Take note that Moz Pro doesn’t affirm if URLs are indexed or identified by Google. Even so, due to the fact most web sites utilize precisely the same robots.txt rules to Moz’s bots as they do to Google’s, this process typically functions effectively for a proxy for Googlebot’s discoverability.

Google Look for Console
Google Look for Console features several beneficial sources for building your listing of URLs.

Links reports:


Comparable to Moz Professional, the Inbound links part supplies exportable lists of goal URLs. Sadly, these exports are capped at one,000 URLs Just about every. It is possible to apply filters for unique internet pages, but due to the fact filters don’t use into the export, you could possibly need to depend upon browser scraping applications—limited to 500 filtered URLs at a time. Not suitable.

Efficiency → Search engine results:


This export gives you a list of webpages getting lookup impressions. Though the export is proscribed, You should use Google Search Console API for bigger datasets. In addition there are cost-free Google Sheets plugins that simplify pulling far more substantial information.

Indexing → Internet pages report:


This portion supplies exports filtered by problem style, while they're also minimal in scope.

Google Analytics
Google Analytics
The Engagement → Internet pages and Screens default report in GA4 is an excellent source for accumulating URLs, which has a generous limit of 100,000 URLs.


Better still, it is possible to implement filters to develop different URL lists, efficiently surpassing the 100k limit. For instance, if you'd like to export only website URLs, stick to these steps:

Move one: Incorporate a segment to the report

Action 2: Simply click “Produce a new phase.”


Stage three: Outline the phase using a narrower URL sample, for example URLs made up of /website/


Take note: URLs found in Google Analytics might not be discoverable by Googlebot or indexed by Google, but they offer useful insights.

Server log documents
Server or CDN log information are Most likely the last word Device at your disposal. These logs capture an exhaustive record of each URL path queried by users, Googlebot, or other bots throughout the recorded interval.

Things to consider:

Knowledge size: Log documents may be massive, countless websites only retain the last two weeks of data.
Complexity: Analyzing log information might be complicated, but various tools are available to simplify the procedure.
Blend, and very good luck
When you finally’ve gathered URLs from all these resources, it’s time to mix them. If your internet site is sufficiently small, use Excel or, for more substantial datasets, tools like Google Sheets or Jupyter Notebook. Assure all URLs are regularly formatted, then deduplicate the checklist.

And voilà—you now have a comprehensive list of recent, outdated, and archived URLs. Fantastic luck!