Let’s be honest: Windows search sucks 😉 You type a file name you know exists, watch the green bar crawl through the Explorer window, and end up with either nothing, the wrong thing, or a web search suggestion you never asked for. The search index covers some folders but not others, it silently skips network drives and external disks, and it has its own idea of what is relevant. When all you want is “where on earth did I put budget_forecast_2019.xlsx ?”, that is just frustrating.
So I went back to an old-school approach: create a plain text file that lists every file of a directory tree once, together with its date and size, and then search that file with grep. Searching a text file with a million lines takes well under a second, works with regular expressions, can be combined with other command-line tools, and gives you exactly what you asked for, nothing more.
Years ago I simply used dir /s > all.txt for that. It works, but the output splits path and file name across lines (so a grep hit on a file name does not tell you where the file is), the format depends on the system language, and special characters in file names turn into codepage garbage. Hence the small Python tool below.
What scandir.py does
scandir.py walks a directory tree and writes one line per file and folder into a text file (default: all.txt). Each line contains the modification date, the size (or <DIR> for folders) and the full path, so every line is self-contained and can be grepped on its own.
- Fast: it uses
os.scandir(), which on Windows gets size and date directly from the directory listing without extra disk I/O. On my machine it indexes about 1.1 million files in 44 seconds. - Same order as
dir /s: entries are sorted case-insensitively, folders and files mixed, and the tree is processed depth-first, with a blank line after each directory listing. - Robust: output is UTF-8; odd file names are written with replacement characters instead of crashing, and folders without access permission are skipped silently.
- Progress bar: it counts the folders on the first two levels beforehand and shows a progress bar on stderr while scanning.
- No dependencies: it needs only the Python standard library; runs on Windows, Linux and macOS.
Usage:
python scandir.py # index the current folder into all.txt
python scandir.py C:\Users\wilfried # index a specific folder
python scandir.py D:\ -o d_drive.txt # index a whole drive into d_drive.txtRun it every now and then (e.g., via the Windows Task Scheduler) to keep the index reasonably current. Since it is just a snapshot, files created after the last run will of course not show up.
Source code
# This script creates a searchable textfile of all files in the file tree.
# The output filename default is all.txt, but it can be changed with the -o option
#
# Takes 44 seconds to index 1.1 million files
#
# 2026 Wil
import os
import sys
import time
ROOT = "."
OUTFILE = "all.txt"
BLOCK_SIZE = 100_000 # lines per write
# Cache formatted timestamps per minute (many files share the same minute)
_date_cache = {}
_strftime = time.strftime
_localtime = time.localtime
def fmt_date(mtime):
key = int(mtime) // 60
s = _date_cache.get(key)
if s is None:
s = _strftime("%d.%m.%Y %H:%M", _localtime(key * 60))
if len(_date_cache) > 200_000:
_date_cache.clear()
_date_cache[key] = s
return s
def sort_key(entry):
return entry.name.upper()
def count_top_dirs(root):
"""Count the folders on levels 1 and 2 below root (used as progress steps)."""
total = 0
try:
with os.scandir(root) as it:
level1 = [e.path for e in it if e.is_dir(follow_symlinks=False)]
except OSError:
return 0
total += len(level1)
for d in level1:
try:
with os.scandir(d) as it:
total += sum(1 for e in it if e.is_dir(follow_symlinks=False))
except OSError:
pass
return total
BAR_W = 40
def show_progress(done, total, state):
# Redraws only when the whole-percent value changes: at most 101 writes per run
pct = 100 if total <= 0 else min(100, done * 100 // total)
if pct != state[0]:
state[0] = pct
filled = BAR_W * pct // 100
sys.stderr.write(f"\r[{'â–ˆ' * filled}{'â–‘' * (BAR_W - filled)}] {pct:3d}%")
sys.stderr.flush()
def main():
root = ROOT
outfile = OUTFILE
args = sys.argv[1:]
i = 0
while i < len(args):
arg = args[i]
low = arg.lower()
if low in ("-o", "--o", "-out", "--out"):
i += 1
if i >= len(args):
sys.exit("missing file name after " + arg)
outfile = args[i]
else:
root = arg
i += 1
root = os.path.abspath(root)
try:
sys.stderr.reconfigure(encoding="utf-8", errors="replace") # bar chars on any console codepage
except (AttributeError, ValueError):
pass
total = count_top_dirs(root)
done = 0
pstate = [-1]
show_progress(0, total, pstate)
stack = [(root, 0)]
buf = []
append = buf.append
scandir = os.scandir
# utf-8 + errors="replace": no codepage garbage, no crash on odd file names
with open(outfile, "w", encoding="utf-8", errors="replace", newline="\n") as f:
write = f.write
while stack:
current, depth = stack.pop()
if 0 < depth <= 2:
done += 1
show_progress(done, total, pstate)
try:
it = scandir(current)
except OSError:
continue
with it:
entries = list(it)
# Same order as `dir /s`: case-insensitive by name, folders and files mixed
entries.sort(key=sort_key)
subdirs = []
start = len(buf)
for entry in entries:
try:
# On Windows, stat() is served from the directory listing: no extra I/O
if entry.is_dir(follow_symlinks=False):
st = entry.stat(follow_symlinks=False)
append(f"{fmt_date(st.st_mtime)}\t<DIR>\t{entry.path}")
subdirs.append((entry.path, depth + 1))
else:
st = entry.stat(follow_symlinks=False)
append(f"{fmt_date(st.st_mtime)}\t{st.st_size}\t{entry.path}")
except OSError:
continue
if len(buf) > start:
append("") # blank line after each directory's listing, like dir /s
# reversed, so the stack pops subfolders in sorted order (depth-first like dir /s)
stack.extend(reversed(subdirs))
if len(buf) >= BLOCK_SIZE:
write("\n".join(buf) + "\n")
buf.clear()
if buf:
write("\n".join(buf) + "\n")
show_progress(total, total, pstate)
sys.stderr.write("\n")
if __name__ == "__main__":
main()Output format
Here is an excerpt of an index generated for a sample directory:
14.03.2018 09:14 <DIR> C:\Users\exampleuser\Documents\webpage\tech.nes.aau.at\sampledir\architecture_12
02.11.2021 14:52 <DIR> C:\Users\exampleuser\Documents\webpage\tech.nes.aau.at\sampledir\infrastructure_70
02.11.2021 14:52 <DIR> C:\Users\exampleuser\Documents\webpage\tech.nes.aau.at\sampledir\q3_reports_29
19.07.2016 22:05 4388 C:\Users\exampleuser\Documents\webpage\tech.nes.aau.at\sampledir\scandir.py
05.05.2023 08:33 1455569 C:\Users\exampleuser\Documents\webpage\tech.nes.aau.at\sampledir\strategy_draft_271.logx
28.09.2019 17:41 <DIR> C:\Users\exampleuser\Documents\webpage\tech.nes.aau.at\sampledir\architecture_12\architecture_41
28.09.2019 17:41 4520241 C:\Users\exampleuser\Documents\webpage\tech.nes.aau.at\sampledir\architecture_12\budget_forecast_791.cfgx
11.01.2015 11:20 <DIR> C:\Users\exampleuser\Documents\webpage\tech.nes.aau.at\sampledir\architecture_12\client_assets_51
11.01.2015 11:20 4452759 C:\Users\exampleuser\Documents\webpage\tech.nes.aau.at\sampledir\architecture_12\client_feedback_415.cfgx
30.06.2025 03:12 2897609 C:\Users\exampleuser\Documents\webpage\tech.nes.aau.at\sampledir\architecture_12\employee_matrix_527.secx
30.06.2025 03:12 <DIR> C:\Users\exampleuser\Documents\webpage\tech.nes.aau.at\sampledir\architecture_12\financials_18
12.12.2017 19:50 7358195 C:\Users\exampleuser\Documents\webpage\tech.nes.aau.at\sampledir\architecture_12\network_topology_537.archEach line has three fields separated by a tab character, date/time, size in bytes (or <DIR> for a folder) and the filename with the full path.
Searching with grep
On Linux and macOS, grep is already there; on Windows you get it with Git for Windows (Git Bash), WSL, or a native port. Some examples:
Find all files with a certain name part (case-insensitive):
grep -i "budget_forecast" all.txt28.09.2019 17:41 4520241 C:\...\sampledir\architecture_12\budget_forecast_791.cfgx
12.12.2017 19:50 1393438 C:\...\sampledir\architecture_12\architecture_41\budget_forecast_744.arch
17.10.2024 16:29 6027244 C:\...\sampledir\architecture_12\client_assets_51\budget_forecast_971.secxFind all files with a given extension (the $ anchors the match at the end of the line):
grep -i "\.cfgx$" all.txtFind only folders whose name contains research_data:
grep "<DIR>" all.txt | grep -i "research_data"Find everything modified in 2026:
grep -E "^[0-9]{2}\.[0-9]{2}\.2026" all.txtFind files larger than 7 MB (here awk uses the tab-separated size column):
awk -F'\t' '$2 ~ /^[0-9]+$/ && $2 > 7000000' all.txtShow only the paths of the hits, without date and size:
grep -i "seminar" all.txt | cut -f3If you don’t have grep at hand, Windows’ built-in findstr also does the job for simple searches: findstr /i "budget_forecast" all.txt. But given the fact that grep is a much more powerful tool, I wamly recommend adding it to your system.
Happy searching!
