How it works

This page explains glep’s design from the ground up: what problem it solves, which constraints it accepts, the mechanism that follows from them, what that mechanism costs, and how it behaves when something goes wrong. The details match the main branch.

The problem

A coding agent calls its Grep and Glob tools dozens of times in one session, over the same tree. ripgrep, the usual engine behind those tools, walks the whole tree and reads every file on every call. That is the right design for one search, and the wrong one for the hundredth search of an unchanged tree.

On the Linux kernel 6.12 source tree (86,604 files, 1.6 GB), one ripgrep full scan took between 1.3 s and 3.5 s on the benchmark machine. At dozens of calls per session, that is minutes of waiting spent re-reading files that cannot match.

The constraints

The project’s own spec states the promise in one line: results must match rg, the index must not lie, and the tool must be safe to call from an agent that does not know where it is standing. With the README’s “no daemon” rule, that gives four constraints:

1. Same bytes as ripgrep. Agents and hooks parse the output. Swapping glep in must change nothing they can see.
2. The index must never lie. A fast answer that misses a match is worse than a slow one, because the agent acts on it.
3. No daemon, no watcher. glep is a command an agent runs. There is no service to start, keep alive, or clean up.
4. Safe from anywhere. The agent may run glep in a subdirectory, pass odd paths, or run several searches in parallel. Each case must still give ripgrep’s answer.

The mechanism

The trigram index

A trigram is three consecutive bytes. When glep indexes a text file it records the set of distinct trigrams the file contains, then inverts that: for each trigram, a sorted list of the files that contain it (a posting list).

Any file that contains the text parse_intent must contain each of its trigrams: par, ars, rse, se_, and so on. Intersecting those posting lists gives every file that could contain the text, plus a few that only contain the pieces. glep then runs the real regex engine over just those candidate files, so the false positives cost a little time and never appear in the output.

The index is file-level, not positional: it says which files, not where in them. It stores raw bytes with no case folding. A case-insensitive search looks up every case variant of each trigram: up to 8 per ASCII trigram, or up to 27 under Unicode folding, where k also matches KELVIN SIGN (U+212A) and s matches LONG S (U+017F).

From a regex to a plan

glep parses the pattern with regex-syntax, ripgrep’s parser, and extracts the literals every match must contain. Adjacent literals fuse into longer strings, literals on both sides of a wildcard are both required, and an alternation becomes an OR of AND-groups. The result is the plan: a list of trigram groups, where a file is a candidate if it contains every trigram of some group. Within a group, glep intersects the rarest posting list first.

Some patterns yield no plan: a literal shorter than three bytes, .*, \w+, an alternation where any branch has no trigram, and any pattern under flags that need every file (-v, --passthru, -c --include-zero, --pre, a non-UTF-8 -E). These search every file. Planning is size-capped (for example, a counted repetition expands at most 64 copies), so a pathological pattern plans in linear time. Each cap only makes the plan narrow less; it never drops a match.

The freshness sweep

Before it answers, every query sweeps the tree: a parallel walk that reads each file’s modification time (mtime) and size, without reading contents, and compares them with the manifest (the index’s file table).

• New and changed files are reindexed into a small delta segment.
• Deleted and replaced entries are marked dead.
• When the delta grows past a tenth of the main postings file, glep rebuilds the whole index (compaction).
• If nothing changed, the query writes nothing (with --ttl set, it only records the sweep time).

The sweep costs one metadata read per file, whatever changed. On macOS it uses getattrlistbulk, one system call per directory. --ttl N skips the sweep when the last full sweep ran within N seconds.

The search

The candidates, plus the files whose contents the index does not hold (over the size cap, unreadable, UTF-16), go to ripgrep’s grep-searcher and are printed with ripgrep’s grep-printer. A pool of worker threads searches them, one searcher per thread, and a printer thread writes results in path order, so the output matches rg --sort path byte for byte while the search runs in parallel. A file up to 1 MiB is read whole into a reused buffer; a larger one is memory-mapped.

Listings

glep --files filters the manifest’s paths by the glob after the sweep. On macOS a root-wide listing uses a directory-mtime cache: creating, deleting or renaming a file changes the modification time of the directory that holds it, so a leaf directory whose mtime is unchanged still holds the same files. Directory mtimes do not propagate upward, so only leaf directories come from the cache; interior directories are always listed live.

Finding the index

glep uses GLEP_INDEX_PATH if it is set. Otherwise it walks up from the current directory to the nearest directory containing .glep/. That directory is the root; the current directory is the search scope, and paths print relative to it. With no index anywhere, the current directory becomes the root.

The first query

On a tree with no index, the first query sweeps the tree once to measure it. A tree holding up to 64 MiB is indexed inline from that sweep. A bigger one is answered from the sweep with a full scan, at ripgrep’s cost, and then indexed by a detached child process at background priority. A query no trigram can narrow builds nothing, since the index would not help it. Windows always builds inline.

The trade-offs

• Disk. The index measured about 10% of the corpus: 154 MB on the kernel tree.
• Build time. A one-time 28 s on the kernel tree, against one ripgrep full scan of 1.3 to 3.5 s. glep needs repeated searches to pay that back.
• Sweep cost on every query. The sweep scales with file count. On the kernel tree a rare pattern took 172 ms with the sweep and 21 ms with --ttl 60.
• Common and unnarrowable patterns gain little. When nearly every file is a candidate, glep does ripgrep’s work plus the sweep. For static on the kernel tree (831k matching lines in 40k files) the published speedup over ripgrep was 1.5x at the lower bound; a single-character pattern ran at ripgrep’s speed.
• Ignored files never enter the index. node_modules, target and other ignored trees stay out, which keeps the index small and the sweep fast. The price is that --no-ignore and the other flags that change which files exist always run a live scan at ripgrep’s cost.

Numbers are from Benchmarks.

Failure modes

• Content changed, metadata did not. The sweep trusts mtime and size. If a file’s contents change while both stay the same (a copy that preserves them, for example), the index keeps the old trigrams, and a search can skip that file. Run glep index to rebuild.
• An edit inside a --ttl window. The sweep is skipped, so the edit is invisible for up to N seconds. Drop --ttl after any edit.
• A corrupt or mismatched index. It is detected on open, deleted and rebuilt, with a note on stderr: glep: index unreadable (...); rebuilding. Each write carries a generation number, so a torn write (manifest and postings from different builds) is detected the same way.
• Several gleps at once. One process writes at a time, under an advisory lock on .glep/lock. Others answer read-only for the tree their own sweep found: deleted files are left out and changed files are searched as they are now. While another glep builds the first index, queries answer with a live scan instead of waiting.
• A macOS fast-path failure. If the bulk sweep fails, glep prints a warning and falls back to the portable walker. A directory containing .ignore or .rgignore also sends the sweep to the portable walker, quietly, because that walker applies their precedence rules exactly.
• A stale listing cache. The cache can only cost speed. When it is missing, stale or too churned, the listing falls back to the full sweep and rebuilds it.
• An index that never persists. On a throwaway continuous integration (CI) runner, every run pays the build. Use ripgrep there.

More symptoms and fixes are in Troubleshooting and limits.