Over on the Fediverse I said something :

It has been '0' days since I've been pointedly reminded¹ that URLs aren't strings.

¹ it wasn't painfully as such because the only thing tripping over it was Bingbot doing bad things, but.

Suppose that you have a URL in the form of a UTF-8 or Unicode text string and you want to show it to people in a web context, and some parts of the URL have characters (or Unicode codepoints) that fall outside of the straightforward 'US-ASCII' character range. If you're putting the URL (string) into HTML, you can often get away with not doing anything special (beyond using UTF-8); your HTML is probably UTF-8 and browsers will do all of the hard work for you. But if you're putting your URL in other places, you may have to encode and quote it. One of those places you need to encode it is in HTTP Location response headers , which you may generate if you're redirecting a URL to some canonical version of it (for example, one without random 'utm' query parameters ).

(There are a few quoting issues in URLs in HTML links and so on that browsers can't sort out for you, but generally you can shovel in any old un-quoted, un-escaped thing and it will probably work out.)

Although browsers may not enforce it, HTTP headers are effectively restricted to US-ASCII or at least ISO-8859-1, which means that you should encode any Unicode or UTF-8 in URLs that appear in Location: (and your web framework may require you to do this). However, Unicode in different parts of the URL must be encoded differently. Unicode in the host portion is encoded using IDNA , while UTF-8 in the path or query parameters is %-encoded. And of course some special characters in the path (especially '?') also need to be %-encoded, while the '?' that separates the path from the query parameters can't be encoded at all (and this is also true for URLs being inserted into HTML).

If you treat an entire URL with scheme, host, path, and query parameters (and possibly a fragment identifier) as a string and try to encode it blindly, you will wind up with suffering. That's what happened to me when I tried to fix my problem the simple way and it blew up in my face, leading to my Fediverse post (more or less).

My specific problem was that Bingbot was occasionally making HTTP requests for URLs with random Unicode characters added to the path, directing that request to alternate, non-canonical hostnames for Wandering Thoughts . If Bingbot had asked for these URLs on the canonical hostname, it would have received a 404 response (no URL paths here have UTF-8) and everything would have been fine. But because Bingbot was asking for an alternate name, DWiki started out by doing a HTTP redirection to the canonical hostname and that redirection echoed Bingbot's mangled, UTF-8 encrusted URL back into the Location: header in the response. This caused a (Python 3) string encoding error when the low level response handler couldn't encode the headers to iso-8859-1.

Sidebar: Unicode versus UTF-8 bytes

Suppose that you get a request with a URL path that ends in '%E2%80%99'. After your web framework decodes this, there are two things you can wind up with at the end of the path; you can get three bytes of UTF-8, 0xE2 0x80 0x99, or you can get the Unicode codepoint U+2019 , the character ’. You'll get the former if your framework doesn't think of URL paths as having a character set and simply undoes %-escapes, and you'll get the latter if your framework assumes UTF-8 encoding and turns the UTF-8 bytes into Unicode. (And then depending on your environment, both versions may print to your terminal and appear in logs the same way.)

(This assumes your language has a distinction between 'Unicode strings' and 'UTF-8 strings', which isn't always the case. Python 3 more or less does, and urllib.parse.unquote() defaults to UTF-8 as the character encoding of the URL path when it turns raw bytes into Unicode strings.)

Conversely, it may matter if you start with a Unicode URL or a UTF-8 URL. If you have a Unicode URL, on the one hand you have the correct starting point for hostnames for IDNA (which is not an encoded representation of UTF-8), but on the other hand you have to decide what character encoding your URL path and query parameters will be in (you'll probably pick UTF-8). If you have a UTF-8 URL, you can decide to directly %-encode the path and so on from your existing bytes, but you need to do something more involved for non-ASCII hostnames to produce the correct IDNA result.