Gobuster: Finding What the Web Server Isn't Supposed to Show You

Series: The Community's Red Team

Post: 03 of 17

Tags: gobuster, web enumeration, directory brute force, dns, vhost, tools

Read time: ~10 minPrerequisites: Post 01 — Methodology Overview, Post 02 — Nmap

Every web server has two versions of itself. The version it shows you the homepage, the login page, the public-facing content. And the version it's hiding the admin panel, the backup files, the staging environment that never got taken down, the config file that ended up in the wrong directory.

Gobuster finds the second version.

It does this by brute force: it takes a wordlist of common directory and file names, fires them at the server one by one, and reports back everything that returns something other than a 404. It's not subtle, it's not stealthy, but in a lab environment or an authorized engagement where noise is acceptable, it is extremely effective. The things it finds such as exposed admin panels, backup zip files, leftover phpinfo.php pages are the kinds of misconfigurations that end engagements in the first thirty minutes.

Three Modes, Three Problems

Gobuster has three modes you'll actually use. Each one solves a different enumeration problem.

dir — brute forces directories and files on a web server. The most common use case. You're looking for paths the server serves but doesn't link to.

dns — brute forces subdomains of a domain. You already know the main domain; you're looking for what's running on dev., staging., admin., vpn., and so on.

vhost — brute forces virtual hosts on a specific IP. Multiple different web applications can run on the same IP address and only respond when called by a specific hostname. Vhost fuzzing finds the ones that aren't advertised.

These aren't interchangeable. A dir scan won't find hidden subdomains. A dns scan won't find hidden directories. Know which problem you're solving.

Mode 1 — Directory and File Brute Force

This is where you start on any target with a web server.

Basic scan

gobuster dir -u http://<target> -w /usr/share/seclists/Discovery/Web-Content/common.txt

-u is the target URL. -w is the wordlist. That's the minimum. common.txt is a fast wordlist that covers the most frequently found paths — good for a first pass.

With file extensions

gobuster dir -u http://<target> \
  -w /usr/share/seclists/Discovery/Web-Content/common.txt \
  -x php,html,txt,bak,zip

The -x flag tells Gobuster to append each extension to every word in the list. So config becomes config.php, config.html, config.txt, config.bak, and config.zip. This matters because a lot of high-value finds are files, not directories — and without -x you'll miss them entirely.

The extensions worth including: php,html,txt,bak,zip,conf,log,sql. Adjust based on what technology the server is running (you got that from Nmap's -sV output or from curl -I).

Full recommended scan

gobuster dir -u http://<target> \
  -w /usr/share/seclists/Discovery/Web-Content/raft-medium-directories.txt \
  -x php,html,txt,bak,zip \
  -t 50 \
  -o dir_output.txt \
  --no-error

-t 50 sets 50 threads — speeds things up significantly without hammering the server. -o saves results to a file so you can reference them later. --no-error suppresses the error spam that shows up when the server resets connections.

Dealing with false positives

Sometimes a server returns 200 for everything — it has a custom 404 page that sends a 200 status. Every path looks open. Gobuster will flood you with results that mean nothing.

Fix it by filtering on status codes:

gobuster dir -u http://<target> -w <wordlist> -b 404,403,301

Or by filtering on response size — if every fake hit is 1,548 bytes, tell Gobuster to ignore that size:

gobuster dir -u http://<target> -w <wordlist> --exclude-length 1548

Find the magic number by running a quick test against a path that definitely doesn't exist and checking the response length in the output.

Wordlist Selection

The wordlist is more important than most people realize. A fast wordlist on a poorly maintained server will find more than a massive wordlist on a well-configured one, but choosing the right wordlist for the situation saves time.

common.txt                        — Fast, hits most common paths. Start here.
raft-medium-directories.txt       — More thorough, good second pass
directory-list-2.3-medium.txt     — Comprehensive, slower
/usr/share/dirb/wordlists/common.txt  — Alternative fast list

All of these are in SecLists — if they're not on your machine:

sudo apt install seclists

Mode 2 — DNS Subdomain Brute Force

You have a domain. You want to know everything running under it.

gobuster dns -d <domain> \
  -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-5000.txt

-d is the target domain. The wordlist here is a list of common subdomain names — dev, staging, admin, vpn, mail, api, git, jenkins, and thousands more.

More thorough scan

gobuster dns -d <domain> \
  -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-20000.txt \
  -t 50 \
  -o dns_output.txt

Custom DNS resolver

In CTF environments, the target's DNS server might know about internal hostnames that your system's resolver doesn't:

gobuster dns -d <domain> -w <wordlist> -r <dns_server_ip>

This forces Gobuster to ask that specific DNS server for resolution — useful when the target is running its own internal DNS.

After finding subdomains, add every single one to /etc/hosts and run a separate scan on each:

echo "<ip> <subdomain>.<domain>" | sudo tee -a /etc/hosts

Mode 3 — Virtual Host Brute Force

This is the mode most beginners skip, and it's where a lot of the interesting content hides.

A virtual host is when one IP address runs multiple completely different web applications, each responding to a different hostname. The server checks the Host: header in the HTTP request and serves different content based on what it sees. A request for admin.target.htb might reach a totally different application than target.htb even though they're the same IP.

Without vhost fuzzing, you'd never know those other applications exist.

Setup

You need the base domain in /etc/hosts first:

sudo sh -c "echo '<ip> <domain>' >> /etc/hosts"

Run the scan

gobuster vhost -u http://<domain> \
  -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-20000.txt \
  --append-domain

--append-domain is required in modern versions of Gobuster. It appends the base domain to each word so admin becomes admin.target.htb in the Host: header. Without it you'll get garbage results.

Filtering false positives

If every vhost returns the same response size, filter it out:

gobuster vhost -u http://<domain> \
  -w <wordlist> \
  --append-domain \
  --exclude-length <size_of_default_response>

Run the scan once, look at the size of the repeated hits, then re-run with that size excluded.

The full vhost workflow

# 1. Add base domain
sudo sh -c "echo '<ip> target.htb' >> /etc/hosts"

# 2. Fuzz for vhosts
gobuster vhost -u http://target.htb -w <wordlist> --append-domain

# 3. Add discovered vhosts
sudo sh -c "echo '<ip> admin.target.htb' >> /etc/hosts"

# 4. Run dir scan on each new vhost
gobuster dir -u http://admin.target.htb -w <wordlist> -x php,html,txt

# 5. Fuzz for nested vhosts (this is common in lab environments)
gobuster vhost -u http://admin.target.htb -w <wordlist> --append-domain

Step 5 is the one most people miss. A discovered vhost is its own web server and might have sub-vhosts of its own. Always scan what you find.

Reading the Output

A Gobuster result looks like this:

/admin                (Status: 301) [Size: 312] [--> http://target/admin/]
/backup               (Status: 200) [Size: 45823]
/config.php           (Status: 200) [Size: 0]
/phpinfo.php          (Status: 200) [Size: 77943]
/.git                 (Status: 301) [Size: 308]

What each result means:

301 redirect — the path exists, server is redirecting to it. Follow the redirect manually and enumerate it.

200 with a large file size — something is there. Open it. /backup returning 45KB is probably an actual backup file.

200 with zero bytes — interesting. Either an empty file or a script that produces no output at your current access level. Try accessing it directly and look at the full response.

phpinfo.php or info.php — jackpot. Contains the full server configuration: PHP version, loaded modules, environment variables, server paths. Often left on servers by developers and forgotten.

.git directory — also jackpot. An exposed git repository means you can potentially download the entire source code of the application, including commit history, which often contains credentials that were committed and later "deleted."

Common High-Value Finds

These are the things that show up in real engagements and CTFs repeatedly:

/admin, /administrator, /admin.php    — Admin panels
/backup, /backup.zip, /backup.tar.gz  — Backup archives
/phpinfo.php, /info.php               — Server info pages  
/config.php, /config.bak, /.env       — Configuration files
/login, /login.php, /wp-login.php     — Login pages
/.git, /.svn                          — Version control directories
/api, /api/v1, /api/v2                — API endpoints
/dev, /staging, /test                 — Development environments
/uploads                              — File upload directories
/robots.txt                           — (read this manually too)

When you find any of these, stop and investigate before continuing the scan. A backup zip file might contain the database. An exposed .git directory might contain hardcoded credentials. The scan can finish while you're exploring.

Common Flags Reference

Flag	What it does
-u	Target URL
-w	Wordlist path
-x	File extensions to append
-t	Threads (default 10, use 50 for speed)
-o	Output file
-b	Blacklist status codes (exclude from results)
-s	Whitelist status codes (only show these)
--exclude-length	Filter results by response size
--no-error	Suppress connection error messages
-k	Skip TLS certificate verification (for HTTPS targets)
-H	Add custom header (-H "Cookie: session=abc")
-c	Add cookie
-r	Custom DNS resolver (dns mode)
-d	Target domain (dns mode)
--append-domain	Append base domain to wordlist entries (vhost mode)

Quick Reference

# Dir scan — fast first pass
gobuster dir -u http://<target> -w /usr/share/seclists/Discovery/Web-Content/common.txt -x php,txt,html,bak,zip -t 50 -o dir.txt

# Dir scan — thorough
gobuster dir -u http://<target> -w /usr/share/seclists/Discovery/Web-Content/raft-medium-directories.txt -x php,txt,html,bak,zip -t 50 -o dir_thorough.txt

# DNS — subdomain discovery
gobuster dns -d <domain> -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-5000.txt -t 50

# DNS — thorough
gobuster dns -d <domain> -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-20000.txt -t 50 -o dns.txt

# Vhost — virtual host discovery
gobuster vhost -u http://<domain> -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-20000.txt --append-domain --exclude-length <default_size> -t 60

# HTTPS target (skip cert verification)
gobuster dir -u https://<target> -k -w <wordlist> -x php,txt,html

# Add found host to /etc/hosts
echo "<ip> <hostname>" | sudo tee -a /etc/hosts

What's Next

If Gobuster found a web application worth exploring further — especially one with a login page, file upload, or unusual response behavior — the next step is Nikto (Post 04), which automatically scans the web server for known vulnerabilities, misconfigurations, and exposed sensitive files.

If you found a login page, jump ahead to Post 07 (Hydra) for brute forcing credentials.

If Nmap showed SMB open alongside the web server, run both paths in parallel — skip to Post 05 (Enum4linux) while Gobuster finishes its scan.

MeshForge — Training the Community's Red Team

They count on your ignorance. The exploit only works on the uninformed.