Mastering Procedural Dungeon Generation in Unity: A Complete Guide

Imagine playing a game where every time you enter a cave, the layout is different. The hallways twist in new directions, the treasure chests hide in different corners, and the exit is never where you last found it. This isn’t magic; it is Procedural Content Generation (PCG). From the endless depths of Diablo to the sprawling galaxies of No Man’s Sky, PCG is the secret sauce that provides infinite replayability without requiring a thousand human designers to build every single room by hand.

For many developers, the jump from hand-placing tiles to writing an algorithm that “thinks” like a level designer feels daunting. You might struggle with “floating” rooms that aren’t connected to anything, or perhaps your code generates messy, unplayable blobs of walls. This guide is designed to bridge that gap. We will move from basic concepts to advanced algorithms, ensuring you leave with a robust system for generating dungeons in Unity.

What is Procedural Dungeon Generation?

At its core, procedural dungeon generation is the use of algorithms to create level layouts automatically. Instead of saving a static scene file, you save a set of rules. When the game starts, the computer follows these rules to carve out floors, place walls, and link them together.

Why should you care?

• Replayability: Players can experience your game hundreds of times without seeing the same layout twice.
• Scalability: You can create massive worlds with a small team.
• Storage: Instead of storing huge level files, you only need to store a “Seed” (a simple string or number) that tells the algorithm how to recreate that specific world.

Understanding the Foundation: The Seed

Before we write a single line of logic, we must understand the Seed. In computer science, “random” numbers are usually “pseudo-random.” They are generated by a mathematical formula that starts with a seed value. If you use the same seed, you get the exact same sequence of numbers.

In game dev, this is a superpower. If a player finds a “cool” map, they can share their seed with a friend, and that friend will see the exact same layout. In Unity, we set this using Random.InitState(seed);.

Method 1: The Random Walker (The “Drunkard’s Walk”)

The simplest way to start is the “Drunkard’s Walk” algorithm. Imagine a person standing in the middle of a grid. They take one step in a random direction (Up, Down, Left, or Right), “carve” a floor tile there, and repeat the process for 1,000 steps. The result is an organic, cave-like structure.

Step-by-Step Implementation

1. Define a starting point in a 2D grid.
2. Choose a random direction.
3. Move to the new position and mark it as “Floor.”
4. Repeat for N iterations.

using System.Collections.Generic;
using UnityEngine;

public class SimpleRandomWalk : MonoBehaviour
{
    public Vector2Int startPosition = Vector2Int.zero;
    public int iterations = 100;
    public int walkLength = 10;
    public bool startRandomlyEachIteration = true;

    // This set will store the coordinates of our floor tiles
    public HashSet<Vector2Int> GeneratePath()
    {
        HashSet<Vector2Int> path = new HashSet<Vector2Int>();
        var currentPosition = startPosition;
        path.Add(currentPosition);

        for (int i = 0; i < iterations; i++)
        {
            // Carve a path for a specific length
            for (int j = 0; j < walkLength; j++)
            {
                // Choose a random direction (Up, Down, Left, Right)
                var newPos = currentPosition + GetRandomDirection();
                path.Add(newPos);
                currentPosition = newPos;
            }

            // If true, the "walker" resets to the start for a new branch
            if (startRandomlyEachIteration)
                currentPosition = startPosition;
        }
        return path;
    }

    private Vector2Int GetRandomDirection()
    {
        Vector2Int[] directions = { Vector2Int.up, Vector2Int.down, Vector2Int.left, Vector2Int.right };
        return directions[Random.Range(0, directions.Length)];
    }
}

Pros: Very easy to code; creates organic, winding paths.
Cons: Often creates messy layouts; no guaranteed “rooms”; can be highly inefficient if the walker keeps overlapping old tiles.

Method 2: Binary Space Partitioning (BSP)

If you want a dungeon that looks like NetHack or Enter the Gungeon (structured rooms and hallways), BSP is the gold standard. BSP works by taking a large rectangular area and repeatedly splitting it into smaller rectangles until you have enough “leaf” nodes to turn into rooms.

How BSP Logic Works:

1. Start with a large rectangle (e.g., 50×50).
2. Split the rectangle horizontally or vertically at a random point.
3. Repeat the split for the resulting rectangles.
4. Once the rectangles reach a minimum size, stop splitting.
5. Shrink each final rectangle slightly to create “rooms” with space for walls.
6. Connect the rooms using hallways.

BSP Room Generation Code

// A helper class to represent a room candidate
public class BoundsIntNode
{
    public BoundsInt bounds;
    public BoundsIntNode leftChild;
    public BoundsIntNode rightChild;

    public BoundsIntNode(BoundsInt bounds)
    {
        this.bounds = bounds;
    }
}

public List<BoundsInt> BinarySpacePartioning(BoundsInt areaToSplit, int minWidth, int minHeight)
{
    Queue<BoundsInt> roomsQueue = new Queue<BoundsInt>();
    List<BoundsInt> roomsList = new List<BoundsInt>();
    roomsQueue.Enqueue(areaToSplit);

    while (roomsQueue.Count > 0)
    {
        var room = roomsQueue.Dequeue();
        if (room.size.y >= minHeight && room.size.x >= minWidth)
        {
            // Logic to decide if we split horizontally or vertically
            if (Random.value > 0.5f)
            {
                if (room.size.y >= minHeight * 2)
                {
                    SplitHorizontally(minHeight, roomsQueue, room);
                }
                else if (room.size.x >= minWidth * 2)
                {
                    SplitVertically(minWidth, roomsQueue, room);
                }
                else
                {
                    roomsList.Add(room);
                }
            }
            else
            {
                if (room.size.x >= minWidth * 2)
                {
                    SplitVertically(minWidth, roomsQueue, room);
                }
                else if (room.size.y >= minHeight * 2)
                {
                    SplitHorizontally(minHeight, roomsQueue, room);
                }
                else
                {
                    roomsList.Add(room);
                }
            }
        }
    }
    return roomsList;
}

private void SplitVertically(int minWidth, Queue<BoundsInt> roomsQueue, BoundsInt room)
{
    var xSplit = Random.Range(1, room.size.x);
    BoundsInt left = new BoundsInt(room.min, new Vector3Int(xSplit, room.size.y, room.size.z));
    BoundsInt right = new BoundsInt(new Vector3Int(room.min.x + xSplit, room.min.y, room.min.z),
        new Vector3Int(room.size.x - xSplit, room.size.y, room.size.z));
    roomsQueue.Enqueue(left);
    roomsQueue.Enqueue(right);
}
// Note: SplitHorizontally follows the same logic on the Y axis

Method 3: Cellular Automata (Organic Caves)

If you are building a game like Terraria or a top-down mining game, you want “blobby,” natural-looking caves. Cellular Automata (often based on Conway’s Game of Life) is the best tool for this.

The Logic:

1. Fill a grid randomly with “Wall” or “Floor” tiles (usually 45% walls).
2. Loop through every tile and count its neighbors (the 8 tiles around it).
3. Rule: If a tile has more than 4 wall neighbors, it becomes a wall. Otherwise, it becomes a floor.
4. Run this process 4 or 5 times to “smooth” the noise into caves.

public int[,] GenerateCave(int width, int height, float fillChance)
{
    int[,] map = new int[width, height];
    
    // Initial random fill
    for (int x = 0; x < width; x++) {
        for (int y = 0; y < height; y++) {
            map[x, y] = (Random.value < fillChance) ? 1 : 0;
        }
    }

    // Smooth the map
    for (int i = 0; i < 5; i++) {
        map = SmoothMap(map, width, height);
    }
    return map;
}

int[,] SmoothMap(int[,] oldMap, int width, int height) {
    int[,] newMap = new int[width, height];
    for (int x = 0; x < width; x++) {
        for (int y = 0; y < height; y++) {
            int neighbors = GetNeighborCount(oldMap, x, y, width, height);
            if (neighbors > 4) newMap[x, y] = 1;
            else if (neighbors < 4) newMap[x, y] = 0;
            else newMap[x, y] = oldMap[x, y];
        }
    }
    return newMap;
}

Connecting Rooms: The Challenge of Accessibility

The biggest mistake in PCG is creating a dungeon where the player can’t reach the end. Whether using BSP or Cellular Automata, you must ensure connectivity.

Ensuring Paths Exist

For BSP, connectivity is easier: since every split happens from a parent rectangle, you can simply draw an L-shaped corridor between the centers of the two child rectangles. Since they were originally one piece, they are guaranteed to be “next” to each other.

For Cellular Automata, you often end up with isolated “pockets” of air. To fix this:

• Use a Flood Fill algorithm to detect all separate “islands” of floor.
• Find the closest points between two islands.
• Dig a tunnel between them.

Unity Implementation: Using Tilemaps

Once your algorithm generates a list of coordinates (the “Data”), you need to render them (the “Visuals”). Unity’s Tilemap system is perfect for this. It handles batching automatically, which is vital for performance.

Rendering Step-by-Step

1. Create a Grid object in your scene.
2. Add a Tilemap and Tilemap Renderer.
3. In your script, reference a TileBase (your wall and floor tiles).
4. Loop through your coordinate set and use tilemap.SetTile(position, tileBase);.

[SerializeField] private Tilemap floorTilemap;
[SerializeField] private TileBase floorTile;

public void PaintFloor(IEnumerable<Vector2Int> floorPositions)
{
    floorTilemap.ClearAllTiles();
    foreach (var pos in floorPositions)
    {
        floorTilemap.SetTile((Vector3Int)pos, floorTile);
    }
}

Common Mistakes and How to Fix Them

1. Using “While” Loops Without Safeguards

When searching for a random spot to place a room, beginners often use: while(!found) { ... }. If the dungeon is full, this creates an Infinite Loop that freezes Unity.

Fix: Always include a counter. int safety = 0; while(!found && safety < 1000) { safety++; ... }.

2. Not Using Object Pooling for Enemies/Loot

If your dungeon has 500 monsters, Instantiate() and Destroy() will cause massive lag spikes as the garbage collector struggles.

Fix: Use Unity’s built-in UnityEngine.Pool to reuse enemy objects.

3. Overly “Pure” Randomness

True randomness is often frustrating. A player might get five empty rooms in a row, then a room with ten bosses.

Fix: Use “Pseudo-Random Distribution” or “Deck Shuffling.” Create a list of what must be in the dungeon (1 boss, 3 chests, 10 grunts), shuffle that list, and pull from it as you generate rooms.

Advanced Topic: The Decorator Pattern

A dungeon of just walls and floors is boring. You need “Decorators.” A Decorator script takes your finished floor layout and runs a second pass to add detail:

• Edge Decorator: Finds floor tiles next to walls and adds “Wall Shadows” or “Torches.”
• Central Decorator: Finds the center of large rooms to place a “Rug” or “Pillar.”
• Constraint Decorator: Ensures the “Exit” is at least 50 units away from the “Spawn.”

Summary / Key Takeaways

• Procedural Generation is about rules, not just randomness. Use Seeds for consistency.
• Use Random Walkers for organic caves and BSP for structured, house-like rooms.
• Cellular Automata is the go-to for natural terrain via the “neighbor-check” method.
• Always separate your Algorithm (Logic) from your Tilemap (Visuals).
• Ensure Connectivity using Flood Fill or center-to-center corridors to prevent soft-locking the player.
• Optimize using Tilemaps and Object Pooling.

Frequently Asked Questions (FAQ)