There are numerous explanations you could want to discover each of the URLs on an internet site, but your precise aim will figure out Whatever you’re trying to find. As an example, you may want to:

Identify every indexed URL to investigate difficulties like cannibalization or index bloat
Accumulate recent and historic URLs Google has observed, especially for site migrations
Obtain all 404 URLs to Get well from submit-migration mistakes
In each state of affairs, an individual Instrument won’t Provide you anything you may need. Sad to say, Google Lookup Console isn’t exhaustive, along with a “site:case in point.com” lookup is limited and tricky to extract knowledge from.

On this post, I’ll wander you through some applications to develop your URL checklist and right before deduplicating the information using a spreadsheet or Jupyter Notebook, based upon your web site’s size.

Aged sitemaps and crawl exports
Should you’re searching for URLs that disappeared within the Dwell site not long ago, there’s a chance anyone with your team could possibly have saved a sitemap file or perhaps a crawl export ahead of the modifications were being created. If you haven’t previously, check for these information; they can typically provide what you require. But, in the event you’re examining this, you most likely didn't get so lucky.

Archive.org
Archive.org
Archive.org is a useful Software for Web optimization tasks, funded by donations. In case you search for a website and choose the “URLs” choice, you may access approximately ten,000 stated URLs.

However, there are a few limitations:

URL limit: You are able to only retrieve around web designer kuala lumpur 10,000 URLs, that's insufficient for more substantial internet sites.
Excellent: Quite a few URLs may very well be malformed or reference useful resource documents (e.g., photographs or scripts).
No export selection: There isn’t a created-in strategy to export the listing.
To bypass the lack of an export button, utilize a browser scraping plugin like Dataminer.io. On the other hand, these constraints indicate Archive.org might not offer an entire Alternative for much larger internet sites. Also, Archive.org doesn’t reveal no matter whether Google indexed a URL—but if Archive.org found it, there’s a very good probability Google did, way too.

Moz Pro
Though you would possibly generally use a connection index to search out external web pages linking for you, these tools also find URLs on your website in the procedure.


The way to utilize it:
Export your inbound back links in Moz Professional to obtain a quick and simple listing of goal URLs from a web site. For those who’re handling a massive Internet site, think about using the Moz API to export info further than what’s manageable in Excel or Google Sheets.

It’s imperative that you Observe that Moz Pro doesn’t verify if URLs are indexed or found out by Google. Even so, due to the fact most web pages utilize a similar robots.txt procedures to Moz’s bots since they do to Google’s, this process commonly will work very well being a proxy for Googlebot’s discoverability.

Google Lookup Console
Google Search Console offers several important resources for constructing your list of URLs.

Backlinks reviews:


Similar to Moz Professional, the Back links area offers exportable lists of target URLs. Regretably, these exports are capped at one,000 URLs Each and every. You'll be able to apply filters for unique web pages, but given that filters don’t apply into the export, you would possibly ought to depend on browser scraping tools—restricted to 500 filtered URLs at any given time. Not excellent.

Efficiency → Search Results:


This export will give you a summary of pages obtaining lookup impressions. While the export is restricted, You can utilize Google Search Console API for much larger datasets. In addition there are no cost Google Sheets plugins that simplify pulling far more in depth details.

Indexing → Web pages report:


This section delivers exports filtered by concern style, however they are also minimal in scope.

Google Analytics
Google Analytics
The Engagement → Webpages and Screens default report in GA4 is a wonderful source for amassing URLs, using a generous Restrict of 100,000 URLs.


A lot better, you are able to utilize filters to produce distinctive URL lists, efficiently surpassing the 100k Restrict. As an example, if you would like export only website URLs, abide by these measures:

Move 1: Insert a segment on the report

Step 2: Simply click “Create a new section.”


Move 3: Outline the phase using a narrower URL sample, like URLs that contains /web site/


Be aware: URLs located in Google Analytics may not be discoverable by Googlebot or indexed by Google, but they offer precious insights.

Server log files
Server or CDN log information are Most likely the ultimate Device at your disposal. These logs seize an exhaustive record of every URL path queried by people, Googlebot, or other bots throughout the recorded time period.

Criteria:

Knowledge measurement: Log files is usually significant, numerous internet sites only keep the last two weeks of information.
Complexity: Analyzing log documents is usually difficult, but a variety of applications are available to simplify the process.
Merge, and very good luck
As you’ve gathered URLs from each one of these sources, it’s time to mix them. If your internet site is sufficiently small, use Excel or, for larger datasets, resources like Google Sheets or Jupyter Notebook. Guarantee all URLs are continually formatted, then deduplicate the record.

And voilà—you now have an extensive listing of current, previous, and archived URLs. Very good luck!