Skip to content
glep

glep / Guides

Indexing and freshness

How glep finds and builds its trigram index, keeps it fresh on every query, which files it skips, and how --ttl and --max-filesize change that.

On this page
  1. First principles
  2. Where the index lives
  3. What gets indexed
  4. The index directory
  5. The first build
  6. Self-healing on every query
  7. Skipping the sweep with –ttl
  8. Raising the size cap
  9. The listing cache (macOS)
  10. Rebuilding and recovery
  11. Seeing where time goes

This page covers where the index lives, what goes into it, how every query keeps it current, and what to do when it goes wrong. It describes glep 2.0.0; release 0.3.1 differs in places listed in Upgrading from 0.3.1.

First principles

  • A trigram index can only answer one question: which files might contain these bytes. It is useful only while it agrees with the files on disk.
  • glep checks that agreement with metadata, not contents. Reading every file to compare would cost a full scan, which is what the index exists to avoid. A file’s modification time (mtime) and size come from one metadata call, or on macOS from one call per directory.
  • What never enters the index cannot go stale in it. Ignored files, files reached through symbolic links, and files on other filesystems stay out, which is why the flags that need them bypass the index entirely.

Where the index lives

glep looks for the index in this order:

  1. GLEP_INDEX_PATH, if set. It points at the index directory itself (for example /work/repo/.glep); its parent is the root.
  2. The nearest directory, starting at the current one and walking up, that contains .glep/.
  3. Otherwise the current directory, where the index will be built.

Run from a subdirectory of an indexed tree, a query searches only that subdirectory and prints paths relative to it, as rg does. glep index run from a subdirectory rebuilds the root’s index.

What gets indexed

glep walks the tree in parallel with ripgrep’s ignore crate, so ignore rules follow ripgrep’s:

  • .gitignore files are honored at every level, whether or not the directory is a git repository. This differs from ripgrep, which applies .gitignore only inside a repository unless you pass --no-require-git. Pass --require-git for ripgrep’s behavior.
  • .ignore and .rgignore files apply too, with .rgignore rules outranking .ignore and .gitignore at any depth, as in ripgrep.
  • .git/info/exclude, .gitignore files in parent directories, and global git excludes (core.excludesFile, $XDG_CONFIG_HOME/git/ignore, ~/.config/git/ignore) also apply.
  • Hidden (dot-prefixed) files and directories are indexed but flagged. They are left out of results unless you pass --hidden, or a ! whitelist rule in an ignore file rescues them.
  • .git and .glep are always excluded, at any depth, even with --hidden.

Each file is classified when it is indexed:

  • Text files have every distinct 3-byte sequence (trigram) recorded.
  • Binary files, with a NUL byte in their first 64 KiB, are flagged and their contents are not indexed. As in ripgrep, they produce no output by default. -a, --binary, --null-data, a non-UTF-8 -E, and --pre make them searchable.
  • UTF-16 files with a byte-order mark are not indexed, because raw bytes cannot narrow decoded text. They are always searched and transcoded first, as in ripgrep.
  • Files larger than the size cap (default 1,048,576 bytes, set with --max-filesize) are flagged and not indexed. They are scanned live on every content query, so results stay complete. Unreadable files are treated the same way.

The index directory

.glep/
  .gitignore     # contains "*", so git ignores the whole directory
  lock           # advisory lock file for the single writer
  manifest.bin   # file table: path, id, mtime, size, flags
  postings.bin   # trigram to file-id lists, memory-mapped
  delta.bin      # recent changes, present after incremental updates
  dirtree.bin    # macOS only: the directory-mtime listing cache

On the Linux kernel tree used for Benchmarks, the index measured 154 MB, about 10% of the corpus.

The first build

You can build the index explicitly with glep index. Otherwise the first query on an unindexed tree builds it, in one of three ways:

  • Small tree (up to 64 MiB of files). The query builds the index inline from the sweep that measured the tree, then answers from it.
  • Large tree. The query answers with a full scan, at ripgrep’s cost, then starts the build in a detached child process at background priority (the utility quality-of-service class on macOS, nice 10 on other Unix systems). The child’s output goes to the null device, so a pipe reading the query is not held open, and Ctrl-C at the terminal leaves it alone. Queries that arrive during the build answer from their own sweep; the first query after it uses the index.
  • A pattern no trigram can narrow. The query searches the tree like --no-index and builds nothing. The first query the index can narrow builds it.

Two environment variables change this. GLEP_SYNC_BUILD=1 always builds inline. GLEP_BG_BUILD_MIN_BYTES moves the 64 MiB threshold and accepts K, M and G suffixes. On Windows the build is always inline.

Self-healing on every query

Before answering, every query runs a freshness sweep: a parallel walk that compares each file’s mtime and size with the manifest.

  • New and changed files are reindexed into a small delta segment.
  • Deleted and replaced entries are marked dead.
  • When the delta grows past a tenth of the main postings file, glep rebuilds the whole index.
  • If nothing changed, the query writes nothing (with --ttl set, it only records the sweep time).
  • A file’s hidden flag follows the ignore files on every sweep, so adding or removing a ! rule that rescues a dotfile takes effect at once.

A query with path operands, or run from a subdirectory, sweeps only those subtrees. Only a full sweep advances the --ttl clock or triggers compaction.

A file counts as changed when its mtime or size differs from the manifest. If a file’s content changes while both stay the same (for example, a copy that preserves mtime and size), the index keeps the old content until the file changes again. Run glep index to rebuild in that case.

The sweep’s cost scales with the number of files, not with how much changed, and on large trees it dominates warm-query time. On macOS the sweep uses getattrlistbulk, one system call per directory instead of one per file. Set GLEP_NO_BULK_SWEEP to force the portable walker. If the fast path fails, glep prints a warning and falls back to the portable walker. A directory containing .ignore or .rgignore also sends the sweep to the portable walker, without a warning.

Skipping the sweep with –ttl

glep --ttl 5 'pattern'

--ttl N skips the freshness sweep if the last recorded full sweep ran within the past N seconds. Scoped queries inside the window skip theirs too. The default is 0: always sweep, always fresh.

The sweep time is recorded by a build, by a sweep that changes the index, and by a full sweep run with --ttl set. A query without --ttl that finds nothing changed does not record it, so a burst’s first --ttl query sweeps and starts the window.

Use it for read-only exploration bursts. Drop it after any file edit, or the edit may be invisible for up to N seconds.

Raising the size cap

glep --max-filesize 2M 'pattern'

The value is a byte count with an optional K, M or G suffix (powers of 1024). The cap applies when a file is indexed. Files already in the index keep their classification until they change, so to apply a new cap to the whole tree, rebuild with it:

glep --max-filesize 2M index

The listing cache (macOS)

On macOS, glep --files over the whole tree uses a directory-mtime cache in .glep/dirtree.bin. Creating, deleting or renaming a file changes the modification time of the directory holding it, so a leaf directory whose mtime matches the cache still holds the same files. The listing walks interior directories live and takes unchanged leaf directories from the cache.

A content query no trigram can narrow, over the whole tree and without --ttl, uses the same cache to get its file list instead of loading the index and sweeping. A missing, stale or corrupt cache only costs speed: the listing falls back to the full sweep and rebuilds it. Set GLEP_NO_DIRCACHE=1 to turn the cache off.

Rebuilding and recovery

  • glep index rebuilds the index from scratch at any time.
  • A corrupt or mismatched index is detected on open, deleted and rebuilt automatically, with a note on stderr: glep: index unreadable (...); rebuilding.
  • One process writes at a time. If another glep holds the lock, a query answers read-only for the tree its own sweep found: deleted files are left out and changed files are searched as they are now. glep index fails with glep: another glep holds the index lock in that case.
  • If another glep is building the first index, queries answer with a live scan instead of waiting.

Seeing where time goes

Set GLEP_TIMING to print per-stage timings to stderr:

GLEP_TIMING=1 glep 'pattern' > /dev/null
glep timing: search path small (2 files)
glep timing: plan 0.1ms
glep timing: index_open 0.1ms
glep timing: sweep_walk 2.0ms
glep timing: sweep_diff 0.1ms
glep timing: index_write 0.0ms
glep timing: operands 0.0ms
glep timing: candidates 0.0ms
glep timing: search 0.2ms
glep timing: total 2.5ms

With --ttl inside the window, the sweep_walk, sweep_diff and index_write stages disappear. Other paths print other stages: live_walk for a live scan, first_sweep and bg_spawn for a first query that defers its build, and cache lines on macOS listings.