Punycode in PHP: Convert IDNs with idn_to_ascii()

Internationalised domain names have been around for over twenty years, and PHP applications still trip over them. Usually not with an error. The DNS lookup for bücher.com just finds nothing, a whois server says “not found”, and the app reports a registered domain as available.

What punycode is and why PHP needs it

DNS is ASCII-only. IDNA, Internationalizing Domain Names in Applications, bridges the gap: each label containing non-ASCII characters is encoded with the punycode algorithm from RFC 3492 and prefixed with xn--. bücher becomes xn--bcher-kva.

Browsers and mail clients convert automatically. Your PHP code doesn’t. Anything that talks to DNS or a registry, dns_get_record(), checkdnsrr(), a whois socket or an RDAP request, needs the ASCII form.

Calling idn_to_ascii() and idn_to_utf8() correctly

Both live in the intl extension, and all four parameters matter:

<?php
declare(strict_types=1);

function toAscii(string $domain): string
{
    $ascii = idn_to_ascii($domain, IDNA_NONTRANSITIONAL_TO_ASCII | IDNA_USE_STD3_RULES | IDNA_CHECK_BIDI | IDNA_CHECK_CONTEXTJ, INTL_IDNA_VARIANT_UTS46, $info);
    if ($ascii === false || $info['errors'] !== 0) {
        throw new InvalidArgumentException(sprintf('"%s" is not a valid IDN (error bits %d)', $domain, $info['errors'] ?? -1));
    }
    return $ascii;
}

function toUnicode(string $domain): string
{
    $utf8 = idn_to_utf8($domain, IDNA_NONTRANSITIONAL_TO_UNICODE, INTL_IDNA_VARIANT_UTS46, $info);
    return ($utf8 === false || $info['errors'] !== 0) ? $domain : $utf8;
}

foreach (['bücher.com', 'straße.de', 'Café.NET', 'пример.рф', '日本.jp', 'exa mple.com', '-ü.com'] as $d) {
    try {
        $a = toAscii($d);
        printf("%-14s -> %-24s -> %s\n", $d, $a, toUnicode($a));
    } catch (InvalidArgumentException $e) {
        echo $e->getMessage(), "\n";
    }
}

Output on PHP 8.5 with ICU 78.3 (columns aligned by hand):

bücher.com     -> xn--bcher-kva.com        -> bücher.com
straße.de      -> xn--strae-oqa.de         -> straße.de
Café.NET       -> xn--caf-dma.net          -> café.net
пример.рф      -> xn--e1afmkfd.xn--p1ai    -> пример.рф
日本.jp        -> xn--wgv71a.jp            -> 日本.jp
"exa mple.com" is not a valid IDN (error bits 128)
"-ü.com" is not a valid IDN (error bits 8)

Already-encoded input passes through unchanged and uppercase is folded, so the function is idempotent. Apply it to every input without worrying.

The flags that matter

IDNA_NONTRANSITIONAL_TO_ASCII is the one you can’t skip. Without it, conversion runs in IDNA2003 transitional mode and maps ß to ss:

<?php
declare(strict_types=1);

var_dump(idn_to_ascii('straße.de', 0, INTL_IDNA_VARIANT_UTS46));
// string(10) "strasse.de"
var_dump(idn_to_ascii('straße.de', IDNA_NONTRANSITIONAL_TO_ASCII, INTL_IDNA_VARIANT_UTS46));
// string(16) "xn--strae-oqa.de"

Those are two different domains that can belong to two different people. The same applies to the Greek final sigma and two zero-width joiner characters. On PHP 8.5 IDNA_DEFAULT evaluates to 48, which already includes both non-transitional flags, but set them explicitly so behaviour doesn’t depend on the server’s PHP build.

IDNA_USE_STD3_RULES rejects characters that don’t belong in hostnames, such as spaces and underscores; without it, exa mple.com comes back unchanged and “valid”. IDNA_CHECK_BIDI and IDNA_CHECK_CONTEXTJ enforce rules for right-to-left scripts and joiner characters. They never hurt for form input.

Reading IDNA error bits

After the call, $info holds result, isTransitionalDifferent and errors. errors is a bitmask. These are the bits you’ll see most:

Bit 2 catches people out. Sixty umlauts look like sixty characters but encode to a label well over 63. Length checks must run on the ASCII form, never on the raw input, which is what the regex in validating domain names does.

isTransitionalDifferent is true when transitional and non-transitional processing would disagree, as with straße.de. You can use it to warn users that a lookalike exists.

Spotting homograph domains

IDNs have a dark side. Cyrillic “а” looks identical to Latin “a”, so exаmple.com with a Cyrillic а is a different domain from example.com. In punycode the difference is obvious: xn--exmple-4nf.com. On screen it often isn’t.

intl ships Spoofchecker for this:

<?php
declare(strict_types=1);

$checker = new Spoofchecker();
var_dump($checker->isSuspicious('exаmple.com'));                    // bool(true), Cyrillic а
var_dump($checker->areConfusable('exаmple.com', 'example.com'));    // bool(true)
var_dump($checker->isSuspicious('bücher.com'));                     // bool(false)

Use it wherever people are meant to trust what they see: admin panels, redirect allowlists, support tools. In a domain search box it’s overkill, since you want to allow Cyrillic names there.

When intl is missing

Some shared hosts don’t ship intl, so function_exists('idn_to_ascii') is false. symfony/polyfill-intl-idn fills in the functions and constants in pure PHP. It doesn’t include Spoofchecker, so check what’s actually installed on the server with php -m | grep intl rather than trusting your laptop.

IDNs inside URLs

URLs are where unconverted names cause the most confusing bugs. filter_var('https://bücher.com/path', FILTER_VALIDATE_URL) returns false, while the punycode version https://xn--bcher-kva.com/path passes. Users paste the Unicode form, so a URL field that validates first and converts never will reject perfectly good input.

The fix is to pull the host out, convert it and put the URL back together. parse_url($url, PHP_URL_HOST) returns bücher.com unchanged, so you can run it through toAscii() and then str_replace() the host in the original string, or rebuild the URL from its parts. Validate afterwards. The same applies to webhook targets, OAuth redirect URIs and anything else stored as a URL: keep the ASCII form in storage, because that’s what cURL and every HTTP library will send.

Where conversion belongs in your code

Convert at the boundary to the outside world, display at the boundary to the user:

• Convert to ASCII before every DNS, whois or RDAP call.
• Store the ASCII form as the unique key.
• Show the Unicode form in UIs and emails.
• For email addresses, convert only the domain part; the local part has its own rules, see PHP email validation.

TLDs can be internationalised too, .рф is xn--p1ai. To find the registrable part of a name, use the Public Suffix List in PHP, which lists both spellings.

To check whether an internationalised name is still free, the betadomain search accepts IDNs directly. A .com domain costs €14.99 a year there, a .eu domain €10.99.