Suppose, hypothetically, that you want to have a personal copy of the content of some website that you feel is important (to you). There are perfectly good reasons to want such a copy; websites go away all the time on the Internet, and not everyone is online all of the time. It's generally possible to do this (and it's certainly possible to do this with Wandering Thoughts ), but there's some things the hypothetical you is going to more or less need to do. These things will be work, but that's the difference between successfully getting a personal copy and turning a brute force crawler lose and then getting ratelimited and blocked. It's also the difference between being polite and being rude , and hopefully you care about that.
(With the increasing decay of Internet search engines, you might also want to build your own personal index of useful website content.)
First, you need to work out the URLs for the real content of the website. Many websites of interest have some mixture of real pages and various sorts of indexes and other aggregations of those real pages, and it's not uncommon for the index pages to outnumber the real pages , sometimes vastly. Your personal copy of the website contents doesn't need all of those index pages, you probably don't want them because they'll inflate the size of your copy, and the website itself will probably be unhappy that you're fetching a ton of redundant index pages.
(The amount of index pages varies with site design. Static sites are usually much friendlier than dynamic sites because it's more work to have a lot of index pages in a static site.)
If you're extremely lucky, the website will have an accurate, up to date (XML) sitemap and will put a tag mentioning this in the HTTML <head> of its pages. If you're not so lucky you will have to manually look around to see if it has any particular index pages that you can mine for URLs ( eg ) and then work out what additional links and pages you need to also fetch to get what you consider a full copy (for example, to also get comments or 'talk' pages or the like, or to fetch images used in the web pages). In less friendly cases you'll have to go through a whole collection of category pages to accumulate the URLs.
(It's possible that the website supports paged syndication feeds and you can go back through its syndication feed to collect a full set of initial URLs, but I suspect that's not any more likely than a discoverable sitemap.)
Having accumulated your list of URLs, it's time to start fetching them, respectfully. Respectful fetching means doing two things: working slowly, and having an honest HTTP User-Agent . Working slowly means that getting a full copy will take a significant amount of time, but unless you think the website is going to go away tomorrow, you have that time. By 'slowly' I mean a request rate of one every 30 seconds or every minute, and if you get HTTP 429s or other indications of rate limits, you should slow down, even if you think this is absurdly slow. In my view, an honest HTTP User-Agent admits to what you're doing and optionally names the software you're using to do the fetching, because the web site operator cares much more about why these requests are happening than that you're using curl, wget, or whatever to make them.
( You especially shouldn't pretend to be a regular browser , or directly use a headless one. In these days of aggressive stealth crawlers, that makes you look extremely suspicious and may well get you blocked rapidly.)
Once you start fetching, you should monitor your fetching for problem indicators. Basically anything other than a HTTP 200 success may be a sign that either you have the wrong URLs or that you're in some way not welcome to do what you're doing. Continuing despite a spate of HTTP redirections or HTTP errors isn't particularly useful for your content copying project; you're only going to have to weed the results out of your copy.
(Also, continuing when a website is telling you 'no' is being rude. You're saying that your desires are more important than the website's views, and this generally makes you a certain sort of person .)
What all of this will get you is a personal copy of the website's content, possibly in addition to a skeletal set of index pages that you can use to navigate through it (you collected these pages when you built the initial URL set). It won't get you a complete archive of the website in HTML form that you could stick up somewhere else. A full website archive is a different thing, one that websites may be much more hostile to depending (in part) on how much redundant content you will wind up crawling in order to assemble your 'complete' version.
(Even if what you want is a full archive of everything, including index pages, starting with the important content first gets you the important content if something goes wrong.)
PS: Wandering Thoughts has a sitemap, which I bashed together many years ago to make Google happy and then found it was convenient for testing because it gave me a list of all pages that I really cared about the HTML rendering of. Interested parties can access it by putting a '?sitemap' on any directory URL. It's not (currently) in the HTML <head> of any pages because when I set it up, that wasn't really a thing. Given the modern web environment, I'm not certain I'll ever make it visible in the HTML <head> because I'm not certain I want to hand every abusive crawler a nice obvious map to the juicy bits.
(I have no idea how long it's been since Google accessed the sitemap; I suspect it's been years. But then, I increasingly don't care about Googlebot, although that's another entry.)