{"id":381,"date":"2026-10-20T11:21:44","date_gmt":"2026-10-20T11:21:44","guid":{"rendered":"https:\/\/tech.nes.aau.at\/?p=381"},"modified":"2026-10-20T11:21:44","modified_gmt":"2026-10-20T11:21:44","slug":"scandir-py-a-fast-greppable-file-index-instead-of-windows-search","status":"publish","type":"post","link":"https:\/\/tech.nes.aau.at\/?p=381","title":{"rendered":"scandir.py: a fast, greppable file index instead of Windows search"},"content":{"rendered":"\n

Let’s be honest: Windows search sucks \ud83d\ude09 You type a file name you know<\/em> exists, watch the green bar crawl through the Explorer window, and end up with either nothing, the wrong thing, or a web search suggestion you never asked for. The search index covers some folders but not others, it silently skips network drives and external disks, and it has its own idea of what is relevant. When all you want is “where on earth did I put budget_forecast_2019.xlsx<\/code> ?”, that is just frustrating.<\/p>\n\n\n\n

So I went back to an old-school approach: create a plain text file that lists every<\/em> file of a directory tree once, together with its date and size, and then search that file with grep<\/code>. Searching a text file with a million lines takes well under a second, works with regular expressions, can be combined with other command-line tools, and gives you exactly what you asked for, nothing more.<\/p>\n\n\n\n

Years ago I simply used dir \/s > all.txt<\/code> for that. It works, but the output splits path and file name across lines (so a grep hit on a file name does not tell you where the file is), the format depends on the system language, and special characters in file names turn into codepage garbage. Hence the small Python tool below.<\/p>\n\n\n\n

What scandir.py does<\/h2>\n\n\n\n

scandir.py<\/code> walks a directory tree and writes one line per file and folder into a text file (default: all.txt<\/code>). Each line contains the modification date, the size (or <DIR><\/code> for folders) and the full path<\/strong>, so every line is self-contained and can be grepped on its own.<\/p>\n\n\n\n

    \n
  • Fast:<\/strong> it uses os.scandir()<\/code>, which on Windows gets size and date directly from the directory listing without extra disk I\/O. On my machine it indexes about 1.1 million files in 44 seconds.<\/li>\n\n\n\n
  • Same order as dir \/s<\/code>:<\/strong> entries are sorted case-insensitively, folders and files mixed, and the tree is processed depth-first, with a blank line after each directory listing.<\/li>\n\n\n\n
  • Robust:<\/strong> output is UTF-8; odd file names are written with replacement characters instead of crashing, and folders without access permission are skipped silently.<\/li>\n\n\n\n
  • Progress bar:<\/strong> it counts the folders on the first two levels beforehand and shows a progress bar on stderr while scanning.<\/li>\n\n\n\n
  • No dependencies:<\/strong> it needs only the Python standard library; runs on Windows, Linux and macOS.<\/li>\n<\/ul>\n\n\n\n

    Usage:<\/p>\n\n\n\n

    python scandir.py                    # index the current folder into all.txt\npython scandir.py C:\\Users\\wilfried  # index a specific folder\npython scandir.py D:\\ -o d_drive.txt # index a whole drive into d_drive.txt<\/code><\/pre>\n\n\n\n
    Run it every now and then (e.g., via the Windows Task Scheduler) to keep the index reasonably current. Since it is just a snapshot, files created after the last run will of course not show up.<\/p>\n\n\n\n
    Source code<\/h2>\n\n\n\n
    # This script creates a searchable textfile of all files in the file tree.\n# The output filename default is all.txt, but it can be changed with the -o option\n#\n# Takes 44 seconds to index 1.1 million files\n#\n# 2026 Wil\n\nimport os\nimport sys\nimport time\n\nROOT = \".\"\nOUTFILE = \"all.txt\"\nBLOCK_SIZE = 100_000  # lines per write\n\n# Cache formatted timestamps per minute (many files share the same minute)\n_date_cache = {}\n_strftime = time.strftime\n_localtime = time.localtime\n\n\ndef fmt_date(mtime):\n    key = int(mtime) \/\/ 60\n    s = _date_cache.get(key)\n    if s is None:\n        s = _strftime(\"%d.%m.%Y  %H:%M\", _localtime(key * 60))\n        if len(_date_cache) > 200_000:\n            _date_cache.clear()\n        _date_cache[key] = s\n    return s\n\n\ndef sort_key(entry):\n    return entry.name.upper()\n\n\ndef count_top_dirs(root):\n    \"\"\"Count the folders on levels 1 and 2 below root (used as progress steps).\"\"\"\n    total = 0\n    try:\n        with os.scandir(root) as it:\n            level1 = [e.path for e in it if e.is_dir(follow_symlinks=False)]\n    except OSError:\n        return 0\n    total += len(level1)\n    for d in level1:\n        try:\n            with os.scandir(d) as it:\n                total += sum(1 for e in it if e.is_dir(follow_symlinks=False))\n        except OSError:\n            pass\n    return total\n\n\nBAR_W = 40\n\n\ndef show_progress(done, total, state):\n    # Redraws only when the whole-percent value changes: at most 101 writes per run\n    pct = 100 if total <= 0 else min(100, done * 100 \/\/ total)\n    if pct != state[0]:\n        state[0] = pct\n        filled = BAR_W * pct \/\/ 100\n        sys.stderr.write(f\"\\r[{'\u2588' * filled}{'\u2591' * (BAR_W - filled)}] {pct:3d}%\")\n        sys.stderr.flush()\n\n\ndef main():\n    root = ROOT\n    outfile = OUTFILE\n    args = sys.argv[1:]\n    i = 0\n    while i < len(args):\n        arg = args[i]\n        low = arg.lower()\n        if low in (\"-o\", \"--o\", \"-out\", \"--out\"):\n            i += 1\n            if i >= len(args):\n                sys.exit(\"missing file name after \" + arg)\n            outfile = args[i]\n        else:\n            root = arg\n        i += 1\n    root = os.path.abspath(root)\n    try:\n        sys.stderr.reconfigure(encoding=\"utf-8\", errors=\"replace\")  # bar chars on any console codepage\n    except (AttributeError, ValueError):\n        pass\n    total = count_top_dirs(root)\n    done = 0\n    pstate = [-1]\n    show_progress(0, total, pstate)\n    stack = [(root, 0)]\n    buf = []\n    append = buf.append\n    scandir = os.scandir\n\n    # utf-8 + errors=\"replace\": no codepage garbage, no crash on odd file names\n    with open(outfile, \"w\", encoding=\"utf-8\", errors=\"replace\", newline=\"\\n\") as f:\n        write = f.write\n        while stack:\n            current, depth = stack.pop()\n            if 0 < depth <= 2:\n                done += 1\n                show_progress(done, total, pstate)\n            try:\n                it = scandir(current)\n            except OSError:\n                continue\n            with it:\n                entries = list(it)\n            # Same order as `dir \/s`: case-insensitive by name, folders and files mixed\n            entries.sort(key=sort_key)\n            subdirs = []\n            start = len(buf)\n            for entry in entries:\n                try:\n                    # On Windows, stat() is served from the directory listing: no extra I\/O\n                    if entry.is_dir(follow_symlinks=False):\n                        st = entry.stat(follow_symlinks=False)\n                        append(f\"{fmt_date(st.st_mtime)}\\t<DIR>\\t{entry.path}\")\n                        subdirs.append((entry.path, depth + 1))\n                    else:\n                        st = entry.stat(follow_symlinks=False)\n                        append(f\"{fmt_date(st.st_mtime)}\\t{st.st_size}\\t{entry.path}\")\n                except OSError:\n                    continue\n            if len(buf) > start:\n                append(\"\")  # blank line after each directory's listing, like dir \/s\n            # reversed, so the stack pops subfolders in sorted order (depth-first like dir \/s)\n            stack.extend(reversed(subdirs))\n            if len(buf) >= BLOCK_SIZE:\n                write(\"\\n\".join(buf) + \"\\n\")\n                buf.clear()\n        if buf:\n            write(\"\\n\".join(buf) + \"\\n\")\n    show_progress(total, total, pstate)\n    sys.stderr.write(\"\\n\")\n\n\nif __name__ == \"__main__\":\n    main()<\/code><\/pre>\n\n\n\n
    Output format<\/h2>\n\n\n\n
    Here is an excerpt of an index generated for a sample directory:<\/p>\n\n\n\n
    14.03.2018  09:14\t<DIR>\tC:\\Users\\exampleuser\\Documents\\webpage\\tech.nes.aau.at\\sampledir\\architecture_12\n02.11.2021  14:52\t<DIR>\tC:\\Users\\exampleuser\\Documents\\webpage\\tech.nes.aau.at\\sampledir\\infrastructure_70\n02.11.2021  14:52\t<DIR>\tC:\\Users\\exampleuser\\Documents\\webpage\\tech.nes.aau.at\\sampledir\\q3_reports_29\n19.07.2016  22:05\t4388\tC:\\Users\\exampleuser\\Documents\\webpage\\tech.nes.aau.at\\sampledir\\scandir.py\n05.05.2023  08:33\t1455569\tC:\\Users\\exampleuser\\Documents\\webpage\\tech.nes.aau.at\\sampledir\\strategy_draft_271.logx\n\n28.09.2019  17:41\t<DIR>\tC:\\Users\\exampleuser\\Documents\\webpage\\tech.nes.aau.at\\sampledir\\architecture_12\\architecture_41\n28.09.2019  17:41\t4520241\tC:\\Users\\exampleuser\\Documents\\webpage\\tech.nes.aau.at\\sampledir\\architecture_12\\budget_forecast_791.cfgx\n11.01.2015  11:20\t<DIR>\tC:\\Users\\exampleuser\\Documents\\webpage\\tech.nes.aau.at\\sampledir\\architecture_12\\client_assets_51\n11.01.2015  11:20\t4452759\tC:\\Users\\exampleuser\\Documents\\webpage\\tech.nes.aau.at\\sampledir\\architecture_12\\client_feedback_415.cfgx\n30.06.2025  03:12\t2897609\tC:\\Users\\exampleuser\\Documents\\webpage\\tech.nes.aau.at\\sampledir\\architecture_12\\employee_matrix_527.secx\n30.06.2025  03:12\t<DIR>\tC:\\Users\\exampleuser\\Documents\\webpage\\tech.nes.aau.at\\sampledir\\architecture_12\\financials_18\n12.12.2017  19:50\t7358195\tC:\\Users\\exampleuser\\Documents\\webpage\\tech.nes.aau.at\\sampledir\\architecture_12\\network_topology_537.arch<\/code><\/pre>\n\n\n\n
    Each line has three fields separated by a tab<\/strong> character, date\/time, size in bytes (or <DIR><\/code> for a folder) and the filename with the full path.<\/p>\n\n\n\n
    Searching with grep<\/h2>\n\n\n\n
    On Linux and macOS, grep<\/code> is already there; on Windows you get it with Git for Windows (Git Bash), WSL, or a native port. Some examples:<\/p>\n\n\n\n
    Find all files with a certain name part (case-insensitive):<\/p>\n\n\n\n
    grep -i \"budget_forecast\" all.txt<\/code><\/pre>\n\n\n\n
    28.09.2019  17:41\t4520241\tC:\\...\\sampledir\\architecture_12\\budget_forecast_791.cfgx\n12.12.2017  19:50\t1393438\tC:\\...\\sampledir\\architecture_12\\architecture_41\\budget_forecast_744.arch\n17.10.2024  16:29\t6027244\tC:\\...\\sampledir\\architecture_12\\client_assets_51\\budget_forecast_971.secx<\/code><\/pre>\n\n\n\n
    Find all files with a given extension (the $<\/code> anchors the match at the end of the line):<\/p>\n\n\n\n
    grep -i \"\\.cfgx$\" all.txt<\/code><\/pre>\n\n\n\n
    Find only folders whose name contains research_data<\/code>:<\/p>\n\n\n\n
    grep \"<DIR>\" all.txt | grep -i \"research_data\"<\/code><\/pre>\n\n\n\n
    Find everything modified in 2026:<\/p>\n\n\n\n
    grep -E \"^[0-9]{2}\\.[0-9]{2}\\.2026\" all.txt<\/code><\/pre>\n\n\n\n
    Find files larger than 7 MB (here awk<\/code> uses the tab-separated size column):<\/p>\n\n\n\n
    awk -F'\\t' '$2 ~ \/^[0-9]+$\/ && $2 > 7000000' all.txt<\/code><\/pre>\n\n\n\n
    Show only the paths of the hits, without date and size:<\/p>\n\n\n\n
    grep -i \"seminar\" all.txt | cut -f3<\/code><\/pre>\n\n\n\n
    If you don’t have grep<\/code> at hand, Windows’ built-in findstr<\/code> also does the job for simple searches: findstr \/i \"budget_forecast\" all.txt<\/code>. But given the fact that grep<\/code> is a much more powerful tool, I wamly recommend adding it to your system.<\/p>\n\n\n\n
    Happy searching!<\/p>\n","protected":false},"excerpt":{"rendered":"
    Let’s be honest: Windows search sucks \ud83d\ude09 You type a file name you know exists, watch the green bar crawl through the Explorer window, and end up with either nothing, the wrong thing, or a web search suggestion you never … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[14],"tags":[17,15,21],"class_list":["post-381","post","type-post","status-publish","format-standard","hentry","category-tools","tag-tool","tag-useful","tag-windows"],"_links":{"self":[{"href":"https:\/\/tech.nes.aau.at\/index.php?rest_route=\/wp\/v2\/posts\/381","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tech.nes.aau.at\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tech.nes.aau.at\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tech.nes.aau.at\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/tech.nes.aau.at\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=381"}],"version-history":[{"count":3,"href":"https:\/\/tech.nes.aau.at\/index.php?rest_route=\/wp\/v2\/posts\/381\/revisions"}],"predecessor-version":[{"id":384,"href":"https:\/\/tech.nes.aau.at\/index.php?rest_route=\/wp\/v2\/posts\/381\/revisions\/384"}],"wp:attachment":[{"href":"https:\/\/tech.nes.aau.at\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=381"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tech.nes.aau.at\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=381"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tech.nes.aau.at\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=381"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}