1. This SO commenter doesn't agree with you, though.
2. Your -v log omits the line that shows the encodings in use; also the Python + libs version lines.
3. --no-restrict-filenames is clearly labelled as (default) in the help text
4. A console font with sufficiently wide Unicode support should display the "? in a box" characters.
5. Have you investigated the +U format code for your filename output template?
6. "this problem immediately began when I switched": that seems weird, since yt-dl doesn't filter weird "Unicode characters" unless --restrict-filenames is specified (yt-dl master first applies the +U (NFKC) format, but the result probably gets _ed anyway); maybe the bundled Python version is responsible?
7. While a conversion that identifies any "confusable" Unicode character with its ASCII equivalent would be possible, the linked table is more strict than desirable, eg 1 (one) == l (lower-case el).

In the end, the windows CLI world just isn't ready for these characters. Windows CLI has worked fine with spaces and ampersands for over 20 yrs, but it's still not there for command-line utilities.

You are confusing the shell in use (cmd, powershell, bash, ...) with the console emulator. In your case you are using conhost, which is ancient and has now been obsoleted with Windows Terminal. Give Windows Terminal a try or even use a completely third party Console emulator, both options support the full range of unicode characters and the like. Also, if it doesnt exist already, install a monospaced font with full unicode character support. You are not restricted to conhost or raster font.

In the end, the windows CLI world just isn't ready for these characters. Windows CLI has worked fine with spaces and ampersands for over 20 yrs, but it's still not there for command-line utilities.

You are confusing the shell in use (cmd, powershell, bash, ...) with the console emulator. In your case you are using conhost, which is ancient and has now been obsoleted with Windows Terminal. Give Windows Terminal a try or even use a completely third party Console emulator, both options support the full range of unicode characters and the like. Also, if it doesnt exist already, install a monospaced font with full unicode character support. You are not restricted to conhost or raster font.

That's all very interesting, but in the end, I don't think that's the solution for me.

Enumerating the many use cases where these characters are problematic is beyond the scope of this bug report. I'm tired of having to find unicode fixes for the myriad of programs I have in my workflows. It's eaten upwards of ~20 hours troubleshooting unicode incompatibility issues in my various workflows in 2023 alone.

These characters are problematic.
In many situations.

(Nor am I looking to to perturb my TCC command-line environment that i've been continuously developing my own layer over for 35 years. )

I don't want the characters in my filename, and that is what this bug report is about.

But perhaps it's really a feature request and not a bug report. My bad on that.

2. Your -v log omits the line that shows the encodings in use; also the Python + libs version lines.

Hmm, well, if something was missing, it must not have been produced as part of the output.

4. A console font with sufficiently wide Unicode support should display the "? in a box" characters.

But alas is something I've tried in the past that created other problems, didn't solve other problems, and ended up not being the use case i need for my situations

I just don't want those characters to be in the filenames, and I don't want to manually have to edit them out, which is what I've been doing. Even if they display properly, they are still problems in situations, so I'm still going to be editing them out.

(Not all software on the planet can deal with them, not all software can be replaced, and I do a lot of niche things where sometimes I'm stuck running old software that can't easily be replaced)

5. Have you investigated the +U format code for your filename output template?

I have but it lookd way too complicated and I don't want to regenerate the whole filename template, i like the way it comes out now, I already have things that ingest the filename format as it stands, so I really just want the character substitution, that's all.

There's already character substitution support, it's just too much. Unicode and spaces are not in the same category of problematic filenames. I just want a more gentle mapping that only substitutes the unicode chracters.

This is seeming more like a feature request than a bug report so I'm wishing I'd filed this in the correct place.

6. "this problem immediately began when I switched": that seems weird, since yt-dl doesn't filter weird "Unicode characters" unless --restrict-filenames is specified (yt-dl master first applies the +U (NFKC) format, but the result probably gets _ed anyway); maybe the bundled Python version is responsible?

It's weird indeed. The youtube-dl.exe i used was a standalone exe, no clue about what kind of python bundling might exist. So many potential explanations including me mis-remembering, me suddenly getting interested in music that has more unicode filenames, etc etc. Hard to look back and ever know why that was when it started to become a problem, it just was. Maybe youtube-dl did save them, but not as often? Or maybe they were omitted by weird incompatibility? Who knows at this point, one could only speculate.

7. While a conversion that identifies any "confusable" Unicode character with its ASCII equivalent would be possible, the linked table is more strict than desirable, eg 1 (one) == l (lower-case el).

It does seem like a problem that's been solved umpteen times, and I'm honestly surprised there's not some "oneliner" library out there that would do exactly this automatically. Conversion to and from utf-8 is easy enough, but yea, this is something a bit different than that.

Imagine my horror the first time i saw a colon in a filename (name specifically, not the path) on a windows machine 🤣

See pinned issues #4836, #5014, especially #5014 (comment)

There is a detox utility that you could run in WSL as a batch filename corrector: needs configuration.

The Unicode site offers a file confusables.txt that lists "confusable" Unicode characters and their equivalent as a sequence of non-confusable characters. This file can be processed to generate various mappings.

I generated a function asc_from_confusable() that takes any Unicode character, finds its entry, if any, and, if all the non-confusable characters are ASCII, returns the string of those characters, or otherwise the original character. Such a function might improve filename sanitization, but the mapping table is 180kB.

$ python3.9
Python 3.9.16 (main, Jan 12 2023, 04:51:49) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import confusables
>>> x = '𝟖'
>>> ord(x)
120790
>>> confusables.asc_from_confusable(x)
'8'
>>> 
$

That confusables file is interesting, but it isn't quite what's needed. It's just remapping unicodes to other unicodes. So they're just as unwanted after your function as before, for my use case.

and yea, I made my own filename fixer, but the problem with my own fixer is the same problem as using --replace-in-metadata -- the mappings still have to be added one character a time.

So far I have 2.
I have full alphabets to go.

And 2 places to put each one - my filename fixer, and my wrapper for ytl-dp (though i could skip that, i'd rather have redundancy)

Ugh.
No clue what i'd want japanese characters to go to, other than their english phonetical equivalent, so that at least i get something that i am pronouncing somewhat correctly.

I don't have the language knowledge to do this well but I suppose with enough time and the right wikipedia page...

I'm pretty much wanting a unicode => ascii mapping. I thought it wasn't a big ask but probably it's bigger than I think (most development ends up being 🤣)

try with --compat-options filename-sanitization https://www.reddit.com/r/youtubedl/comments/yz9ozo/how_do_i_get_ytdlp_downloads_without_forbidden/

try with --compat-options filename-sanitization

Didn't change the behavior.

At this point I've done an hour or so work of getting the polyglot library installed on python to detect language and romanize where possible, and then have my own mapping table of characters not caught by that. it's ugly but it's finally starting to automatically rename these bad filenames

It runs the whole filename through polyglot as one string, which may or may not detect language and translate

Then it goes character by character checks specifically for 3 languages I care about and uses 3 language-specific libraries for those chracters

finally it goes through my own mapping table

It's a doozy and i was probably asking too much for this before.... I guess I just solved my own problem.

import os
import sys
import unidecode
from colorama import Fore, Style, init
import re

DRY_RUN = False


# Mapping of unicode symbols to ASCII equivalents that are valid for filenames
unicode_to_valid_filename_ASCII_map = {
    '｜':   '-' ,  # unicode pipe
    '！':   '!' ,  # unicode exclamation mark
    '？':   '_' ,  # unicode question mark
    '：' :   '- ',  # unicode colon
    '；' :   ';' ,  # unicode semicolon
    '，' :   ',' ,  # unicode comma
    '。' :   '.' ,  # unicode full stop
    '⧸' :   '--',  # unicode slash           
}



def translate_character(char):
    """Translates a single character to its ASCII equivalent."""
    if char in unicode_to_valid_filename_ASCII_map: return unicode_to_valid_filename_ASCII_map[char]
    if   '\u4e00' <= char <= '\u9fff': ascii_equiv = translate_chinese_to_ascii (char)            # if Chinese
    elif '\u3040' <= char <= '\u30ff': ascii_equiv = translate_japanese_to_ascii(char)            # if Japanese
    elif '\uac00' <= char <= '\ud7af': ascii_equiv = translate_korean_to_ascii  (char)            # if Korean
    else:                              ascii_equiv = unidecode.unidecode        (char)            # if Unicode
    #eturn ascii_equiv if ascii_equiv.isalnum() else '_'  #BAD: turned " " to "_"
    return ascii_equiv


# Japanese
import romkan
def translate_japanese_to_ascii(char):
    return romkan.to_roma(char)

# Chinese
from pypinyin import lazy_pinyin, Style as PypinyinStyle
def translate_chinese_to_ascii(char):
    return ''.join(lazy_pinyin(char, style=PypinyinStyle.TONE3))

# Korean
from korean_romanizer.romanizer import Romanizer
def translate_korean_to_ascii(korean_text):
    r = Romanizer(korean_text)
    return r.romanize()



def romanize(text):
    """Return translated text, but fail very gracefully and transparently if there are any exceptions"""
    try:
        import logging
        logging.getLogger('polyglot').setLevel(logging.ERROR)   # Disable logging messages from Polyglot
        from polyglot.detect import Detector
        from polyglot.transliteration import Transliterator
        detector = Detector(text)
        source_lang = detector.language.code
        transliterator = Transliterator(source_lang=source_lang, target_lang="en")
        return transliterator.transliterate(text)
    except Exception:
        return(text)



def translate_filename(filename):
    """Translates a filename to its ASCII equivalent."""
    filename_romanized_with_polyglot = romanize(filename)
    return ''.join(translate_character(char) for char in filename_romanized_with_polyglot)

import msvcrt
def ask_permission(old_name, new_name):
    """Asks the user for permission to rename a file."""
    print(f"\n{Fore.YELLOW}{Style.BRIGHT}***** Rename:"                                                                   +
          f"\n{Fore.RED   }{Style.BRIGHT}From: {Style.NORMAL}{old_name}{Fore.CYAN}{Style.NORMAL}"                         +
          f"\n{Fore.GREEN }{Style.BRIGHT}  To: {Style.NORMAL}{new_name}{Fore.CYAN}{Style.NORMAL} "                        +
          f"\n{Fore.YELLOW}{Style.BRIGHT}***** Rename?"                                                                   +
          f" { Fore.BLUE  }{Style.BRIGHT}[{Fore.CYAN}Y{Fore.BLUE}/{Style.NORMAL}{Fore.CYAN}n{Style.BRIGHT}]{Style.NORMAL} ", end="")
    response = msvcrt.getch().decode().lower().strip()
    print(Style.BRIGHT, end="")
    if response.lower() in ['y', 'yes', '']:
        print(f"{Fore.GREEN}Yes!", end="")
        return True
    print(f"{Fore.RED}No!", end="")
    return False

def rename_files_in_directory(directory):
    """Renames all files in a directory, replacing unicode characters."""
    global DRY_RUN
    do_it_for_real = True
    automatic      = False
    DRY_RUN        = False
    permission     = False
    for filename in os.listdir(directory):
        new_name = translate_filename(filename)

        if filename != new_name:
            if       os.getenv('AUTOMATIC_UNICODE_CLEANING'):
                del os.environ['AUTOMATIC_UNICODE_CLEANING']                    #only let this directive work once!
                automatic      = True
                do_it_for_real = True
                action_string  = "  Auto-Renamed"
            else:
                permission = ask_permission(filename, new_name)
                do_it_for_real = permission
                action_string  = "       Renamed" if permission is True else f"{Fore.RED}Did not rename"
            if DRY_RUN:
                do_it_for_real = False

            old_file = os.path.join(directory, filename)
            new_file = os.path.join(directory, new_name)

            if do_it_for_real: os.rename(old_file, new_file)

            print("\n")
            if automatic: print(f"\t{Fore.YELLOW}Automatic Run: ")
            if DRY_RUN:   print(f"\t{Fore.YELLOW}" +  "Dry Run: ")
            print(f"{Fore.GREEN}{Style.NORMAL}\t{action_string}:\t{Fore.LIGHTBLACK_EX}{old_file} " +
                  f"{Fore.CYAN}\n\t\t    to:\t{Fore.GREEN}{new_file}{Style.NORMAL}\n\n\n")

def main():
    init()
    directory = sys.argv[1] if len(sys.argv) > 1 else '.'
    rename_files_in_directory(directory)


if __name__ == "__main__":
    main()

I use -compat filename-sanitization

try with --compat-options filename-sanitization

Didn't change the behavior.

At this point I've done an hour or so work of getting the polyglot library installed on python to detect language and romanize where possible, and then have my own mapping table of characters not caught by that. it's ugly but it's finally starting to automatically rename these bad filenames

It runs the whole filename through polyglot as one string, which may or may not detect language and translate

Then it goes character by character checks specifically for 3 languages I care about and uses 3 language-specific libraries for those chracters

finally it goes through my own mapping table

It's a doozy and i was probably asking too much for this before.... I guess I just solved my own problem.

import os
import sys
import unidecode
from colorama import Fore, Style, init
import re

DRY_RUN = False


# Mapping of unicode symbols to ASCII equivalents that are valid for filenames
unicode_to_valid_filename_ASCII_map = {
    '｜':   '-' ,  # unicode pipe
    '！':   '!' ,  # unicode exclamation mark
    '？':   '_' ,  # unicode question mark
    '：' :   '- ',  # unicode colon
    '；' :   ';' ,  # unicode semicolon
    '，' :   ',' ,  # unicode comma
    '。' :   '.' ,  # unicode full stop
    '⧸' :   '--',  # unicode slash           
}



def translate_character(char):
    """Translates a single character to its ASCII equivalent."""
    if char in unicode_to_valid_filename_ASCII_map: return unicode_to_valid_filename_ASCII_map[char]
    if   '\u4e00' <= char <= '\u9fff': ascii_equiv = translate_chinese_to_ascii (char)            # if Chinese
    elif '\u3040' <= char <= '\u30ff': ascii_equiv = translate_japanese_to_ascii(char)            # if Japanese
    elif '\uac00' <= char <= '\ud7af': ascii_equiv = translate_korean_to_ascii  (char)            # if Korean
    else:                              ascii_equiv = unidecode.unidecode        (char)            # if Unicode
    #eturn ascii_equiv if ascii_equiv.isalnum() else '_'  #BAD: turned " " to "_"
    return ascii_equiv


# Japanese
import romkan
def translate_japanese_to_ascii(char):
    return romkan.to_roma(char)

# Chinese
from pypinyin import lazy_pinyin, Style as PypinyinStyle
def translate_chinese_to_ascii(char):
    return ''.join(lazy_pinyin(char, style=PypinyinStyle.TONE3))

# Korean
from korean_romanizer.romanizer import Romanizer
def translate_korean_to_ascii(korean_text):
    r = Romanizer(korean_text)
    return r.romanize()



def romanize(text):
    """Return translated text, but fail very gracefully and transparently if there are any exceptions"""
    try:
        import logging
        logging.getLogger('polyglot').setLevel(logging.ERROR)   # Disable logging messages from Polyglot
        from polyglot.detect import Detector
        from polyglot.transliteration import Transliterator
        detector = Detector(text)
        source_lang = detector.language.code
        transliterator = Transliterator(source_lang=source_lang, target_lang="en")
        return transliterator.transliterate(text)
    except Exception:
        return(text)



def translate_filename(filename):
    """Translates a filename to its ASCII equivalent."""
    filename_romanized_with_polyglot = romanize(filename)
    return ''.join(translate_character(char) for char in filename_romanized_with_polyglot)

import msvcrt
def ask_permission(old_name, new_name):
    """Asks the user for permission to rename a file."""
    print(f"\n{Fore.YELLOW}{Style.BRIGHT}***** Rename:"                                                                   +
          f"\n{Fore.RED   }{Style.BRIGHT}From: {Style.NORMAL}{old_name}{Fore.CYAN}{Style.NORMAL}"                         +
          f"\n{Fore.GREEN }{Style.BRIGHT}  To: {Style.NORMAL}{new_name}{Fore.CYAN}{Style.NORMAL} "                        +
          f"\n{Fore.YELLOW}{Style.BRIGHT}***** Rename?"                                                                   +
          f" { Fore.BLUE  }{Style.BRIGHT}[{Fore.CYAN}Y{Fore.BLUE}/{Style.NORMAL}{Fore.CYAN}n{Style.BRIGHT}]{Style.NORMAL} ", end="")
    response = msvcrt.getch().decode().lower().strip()
    print(Style.BRIGHT, end="")
    if response.lower() in ['y', 'yes', '']:
        print(f"{Fore.GREEN}Yes!", end="")
        return True
    print(f"{Fore.RED}No!", end="")
    return False

def rename_files_in_directory(directory):
    """Renames all files in a directory, replacing unicode characters."""
    global DRY_RUN
    do_it_for_real = True
    automatic      = False
    DRY_RUN        = False
    permission     = False
    for filename in os.listdir(directory):
        new_name = translate_filename(filename)

        if filename != new_name:
            if       os.getenv('AUTOMATIC_UNICODE_CLEANING'):
                del os.environ['AUTOMATIC_UNICODE_CLEANING']                    #only let this directive work once!
                automatic      = True
                do_it_for_real = True
                action_string  = "  Auto-Renamed"
            else:
                permission = ask_permission(filename, new_name)
                do_it_for_real = permission
                action_string  = "       Renamed" if permission is True else f"{Fore.RED}Did not rename"
            if DRY_RUN:
                do_it_for_real = False

            old_file = os.path.join(directory, filename)
            new_file = os.path.join(directory, new_name)

            if do_it_for_real: os.rename(old_file, new_file)

            print("\n")
            if automatic: print(f"\t{Fore.YELLOW}Automatic Run: ")
            if DRY_RUN:   print(f"\t{Fore.YELLOW}" +  "Dry Run: ")
            print(f"{Fore.GREEN}{Style.NORMAL}\t{action_string}:\t{Fore.LIGHTBLACK_EX}{old_file} " +
                  f"{Fore.CYAN}\n\t\t    to:\t{Fore.GREEN}{new_file}{Style.NORMAL}\n\n\n")

def main():
    init()
    directory = sys.argv[1] if len(sys.argv) > 1 else '.'
    rename_files_in_directory(directory)


if __name__ == "__main__":
    main()

#11046
Can you save me bro

try with --compat-options filename-sanitization

Didn't change the behavior.
At this point I've done an hour or so work of getting the polyglot library installed on python to detect language and romanize where possible, and then have my own mapping table of characters not caught by that. it's ugly but it's finally starting to automatically rename these bad filenames
It runs the whole filename through polyglot as one string, which may or may not detect language and translate
Then it goes character by character checks specifically for 3 languages I care about and uses 3 language-specific libraries for those chracters
finally it goes through my own mapping table
It's a doozy and i was probably asking too much for this before.... I guess I just solved my own problem.

import os
import sys
import unidecode
from colorama import Fore, Style, init
import re

DRY_RUN = False


# Mapping of unicode symbols to ASCII equivalents that are valid for filenames
unicode_to_valid_filename_ASCII_map = {
    '｜':   '-' ,  # unicode pipe
    '！':   '!' ,  # unicode exclamation mark
    '？':   '_' ,  # unicode question mark
    '：' :   '- ',  # unicode colon
    '；' :   ';' ,  # unicode semicolon
    '，' :   ',' ,  # unicode comma
    '。' :   '.' ,  # unicode full stop
    '⧸' :   '--',  # unicode slash           
}



def translate_character(char):
    """Translates a single character to its ASCII equivalent."""
    if char in unicode_to_valid_filename_ASCII_map: return unicode_to_valid_filename_ASCII_map[char]
    if   '\u4e00' <= char <= '\u9fff': ascii_equiv = translate_chinese_to_ascii (char)            # if Chinese
    elif '\u3040' <= char <= '\u30ff': ascii_equiv = translate_japanese_to_ascii(char)            # if Japanese
    elif '\uac00' <= char <= '\ud7af': ascii_equiv = translate_korean_to_ascii  (char)            # if Korean
    else:                              ascii_equiv = unidecode.unidecode        (char)            # if Unicode
    #eturn ascii_equiv if ascii_equiv.isalnum() else '_'  #BAD: turned " " to "_"
    return ascii_equiv


# Japanese
import romkan
def translate_japanese_to_ascii(char):
    return romkan.to_roma(char)

# Chinese
from pypinyin import lazy_pinyin, Style as PypinyinStyle
def translate_chinese_to_ascii(char):
    return ''.join(lazy_pinyin(char, style=PypinyinStyle.TONE3))

# Korean
from korean_romanizer.romanizer import Romanizer
def translate_korean_to_ascii(korean_text):
    r = Romanizer(korean_text)
    return r.romanize()



def romanize(text):
    """Return translated text, but fail very gracefully and transparently if there are any exceptions"""
    try:
        import logging
        logging.getLogger('polyglot').setLevel(logging.ERROR)   # Disable logging messages from Polyglot
        from polyglot.detect import Detector
        from polyglot.transliteration import Transliterator
        detector = Detector(text)
        source_lang = detector.language.code
        transliterator = Transliterator(source_lang=source_lang, target_lang="en")
        return transliterator.transliterate(text)
    except Exception:
        return(text)



def translate_filename(filename):
    """Translates a filename to its ASCII equivalent."""
    filename_romanized_with_polyglot = romanize(filename)
    return ''.join(translate_character(char) for char in filename_romanized_with_polyglot)

import msvcrt
def ask_permission(old_name, new_name):
    """Asks the user for permission to rename a file."""
    print(f"\n{Fore.YELLOW}{Style.BRIGHT}***** Rename:"                                                                   +
          f"\n{Fore.RED   }{Style.BRIGHT}From: {Style.NORMAL}{old_name}{Fore.CYAN}{Style.NORMAL}"                         +
          f"\n{Fore.GREEN }{Style.BRIGHT}  To: {Style.NORMAL}{new_name}{Fore.CYAN}{Style.NORMAL} "                        +
          f"\n{Fore.YELLOW}{Style.BRIGHT}***** Rename?"                                                                   +
          f" { Fore.BLUE  }{Style.BRIGHT}[{Fore.CYAN}Y{Fore.BLUE}/{Style.NORMAL}{Fore.CYAN}n{Style.BRIGHT}]{Style.NORMAL} ", end="")
    response = msvcrt.getch().decode().lower().strip()
    print(Style.BRIGHT, end="")
    if response.lower() in ['y', 'yes', '']:
        print(f"{Fore.GREEN}Yes!", end="")
        return True
    print(f"{Fore.RED}No!", end="")
    return False

def rename_files_in_directory(directory):
    """Renames all files in a directory, replacing unicode characters."""
    global DRY_RUN
    do_it_for_real = True
    automatic      = False
    DRY_RUN        = False
    permission     = False
    for filename in os.listdir(directory):
        new_name = translate_filename(filename)

        if filename != new_name:
            if       os.getenv('AUTOMATIC_UNICODE_CLEANING'):
                del os.environ['AUTOMATIC_UNICODE_CLEANING']                    #only let this directive work once!
                automatic      = True
                do_it_for_real = True
                action_string  = "  Auto-Renamed"
            else:
                permission = ask_permission(filename, new_name)
                do_it_for_real = permission
                action_string  = "       Renamed" if permission is True else f"{Fore.RED}Did not rename"
            if DRY_RUN:
                do_it_for_real = False

            old_file = os.path.join(directory, filename)
            new_file = os.path.join(directory, new_name)

            if do_it_for_real: os.rename(old_file, new_file)

            print("\n")
            if automatic: print(f"\t{Fore.YELLOW}Automatic Run: ")
            if DRY_RUN:   print(f"\t{Fore.YELLOW}" +  "Dry Run: ")
            print(f"{Fore.GREEN}{Style.NORMAL}\t{action_string}:\t{Fore.LIGHTBLACK_EX}{old_file} " +
                  f"{Fore.CYAN}\n\t\t    to:\t{Fore.GREEN}{new_file}{Style.NORMAL}\n\n\n")

def main():
    init()
    directory = sys.argv[1] if len(sys.argv) > 1 else '.'
    rename_files_in_directory(directory)


if __name__ == "__main__":
    main()

#11046 Can you save me bro

I'm not a bro, but check out my fix_unicode_filename projects on my github. It's a more current [and bulky] version of this. Alas, i still have to edit it when a new character i haven't encountered creeps up, so it's far from perfect, but i have it as part of my yt-dlp workflow to do what yt-dlp won't do.

Emoji and unicode screw up a lot of software. My image viewer won't even view images if a folder has an emoji in it, for example.