Skip to content
MCP Filesystem Ultra

Bug #13: smart_search Slow on Large Projects

Status: RESOLVED in v3.14.3 Category: Performance Severity: High (search becomes unusable on real-world projects) Resolution Date: 2026-02-27

Problem

Section titled “Problem”

smart_search and advanced_text_search became extremely slow on any project that contained build output directories (bin/, obj/, node_modules/, .vs/, etc.) or files with extensions not known to the server (.aspx, .cshtml, .resx, etc.).

What Happened

Section titled “What Happened”

A search for BuildShipmentHtml|ShipmentHtml|ExpedicionHtml in a .NET CRM project at C:\__REPOS\...\crm took tens of seconds to complete, or timed out entirely. The project contained standard ASP.NET structure — Razor views, compiled assemblies in bin/ and obj/, Visual Studio cache in .vs/, and NuGet packages in packages/.

Root Cause

Section titled “Root Cause”

Three compounding problems in performSmartSearch and performAdvancedTextSearch:

Problem 1 (worst): validatePath → EvalSymlinks called per file

Inside the filepath.Walk callback, every single file and directory called validatePath:

// BEFORE (broken)
err = filepath.Walk(path, func(currentPath string, info os.FileInfo, err error) error {
    ...
    // Validate path
    if _, err := e.validatePath(currentPath); err != nil {
        return nil
    }
    ...
})

validatePath calls isPathAllowed, which calls filepath.EvalSymlinks — a real I/O syscall that resolves the entire symlink chain. On a project with 8,000 files, this was 8,000 EvalSymlinks calls before a single line of code was searched. The root path is already validated before the walk starts; checking each child file adds zero security benefit.

Problem 2: No directory pruning

Neither walk had a filepath.SkipDir guard. The walk descended into every directory without exception, including:

DirectoryWhy it’s expensive
bin/.NET compiled assemblies — hundreds of .dll, .pdb, .exe files
obj/Intermediate build output — .cs generated files, .cache, .json metadata
.vs/Visual Studio workspace cache — thousands of small binary files
packages/NuGet packages — full source and binaries for every dependency
node_modules/npm packages — often >100k files
.git/Git object store — compressed pack files and loose objects

A medium .NET solution can easily have 20,000+ files in bin/ + obj/ alone.

Problem 3: isTextFile opened unknown-extension files

isTextFile checks a fast O(1) map first, but falls through to reading 512 bytes from disk for any extension not in the map. Common ASP.NET extensions were absent:

Missing extensionFile type
.aspxASP.NET WebForms page
.ascxUser control
.cshtmlRazor view
.razorBlazor component
.resxResource file
.csproj / .slnProject / solution file
.xaml / .axamlXAML markup
.targets / .propsMSBuild files

Every file with one of these extensions triggered a file Open + Read(512) syscall just to be classified as text — before any pattern matching occurred.

Combined Effect

Section titled “Combined Effect”

For a 2,000-file CRM project:

  • 8,000 EvalSymlinks calls (validatePath on each file and directory node)
  • ~5,000 extra file opens for isTextFile fallback on unknown extensions
  • Full traversal of bin/ and obj/ — thousands of binary files that cannot match

Total: the walk spent >90% of its time on overhead, not on searching.

Solution

Section titled “Solution”

Three targeted fixes, each addressing one root cause.

Fix 1: Remove validatePath from walk callback

Section titled “Fix 1: Remove validatePath from walk callback”
// AFTER: root path validated once before the walk; children inherit that grant
err = filepath.Walk(path, func(currentPath string, info os.FileInfo, err error) error {
    if ctxErr := ctx.Err(); ctxErr != nil {
        return ctxErr
    }
    if err != nil {
        return nil
    }


    if info.IsDir() {
        if searchSkipDirs[info.Name()] {
            return filepath.SkipDir  // ← prune here (Fix 2)
        }
        return nil
    }
    // No validatePath call — root was already validated
    ...
})

Fix 2: Prune irrelevant directories with filepath.SkipDir

Section titled “Fix 2: Prune irrelevant directories with filepath.SkipDir”
// searchSkipDirs are directories skipped during search walks.
var searchSkipDirs = map[string]bool{
    // Version control
    ".git": true, ".svn": true, ".hg": true,
    // JS/Node
    "node_modules": true, ".next": true, ".nuxt": true, "dist": true,
    // .NET / Visual Studio
    "bin": true, "obj": true, ".vs": true, "packages": true, ".nuget": true,
    // Java / Maven / Gradle
    "target": true, ".gradle": true,
    // Python
    "__pycache__": true, ".venv": true, "venv": true, ".eggs": true,
    // General build/cache
    "build": true, ".cache": true, ".tmp": true,
}

When filepath.Walk encounters a directory in this set, it returns filepath.SkipDir — the entire subtree is skipped without visiting any of its children.

Fix 3: Add ASP.NET and MSBuild extensions to textExtensionsMap

Section titled “Fix 3: Add ASP.NET and MSBuild extensions to textExtensionsMap”
.NET
".cs": true, ".fs": true, ".vb": true,
".csproj": true, ".vbproj": true, ".fsproj": true, ".sln": true,
".aspx": true, ".ascx": true, ".ashx": true, ".asmx": true, ".asax": true,
".cshtml": true, ".vbhtml": true, ".razor": true,
".resx": true, ".xaml": true, ".axaml": true,
".targets": true, ".props": true, ".nuspec": true,

These extensions now resolve in a single O(1) map lookup instead of opening the file.

Files Changed

Section titled “Files Changed”
FileChange
core/search_operations.goRemoved validatePath from both walk callbacks
core/search_operations.goAdded searchSkipDirs map; both walks return filepath.SkipDir for pruned dirs
core/search_operations.goAdded 14 ASP.NET / MSBuild extensions to textExtensionsMap

Verification

Section titled “Verification”
go build ./...          → OK (no errors)
go test ./tests/...     → PASS
go test ./core/...      → PASS

Lessons Learned

Section titled “Lessons Learned”

  1. Never call I/O inside a walk callback unless strictly necessary. validatePath in the callback looked like a security measure but provided no additional protection — the root was already validated. One EvalSymlinks per file is an O(N) syscall storm.

  2. filepath.SkipDir is essential for real projects. Build artifact directories are structurally identical to source directories from the OS perspective. The walk cannot distinguish them without an explicit exclusion list.

  3. Every unrecognized extension is a disk read. The isTextFile fallback is correct for truly unknown files, but well-known platform extensions should never fall through to it. Adding extensions is cheap; omitting them costs a file open per unknown file.

  4. Test with real project structures, not synthetic directories. A flat directory of 100 .go files shows no performance issue. A real .NET solution with bin/ and obj/ exposes the problem immediately.

Section titled “Related”