An autonomous AI agent that plays Minecraft using LLMs and JavaScript interop.
Created for firs meeting of the Pupunha Code Community
- About The Project
- Features
- Architecture
- Getting Started
- Configuration
- Usage
- Project Structure
- Tools & Capabilities
- Contributing
- License
- Acknowledgments
- Contact
This project was developed as a live-coding demonstration for a talk at Pupunha Code, a vibrant developer community. It showcases how to build an autonomous AI agent capable of playing Minecraft by integrating a powerful stack:
- Python: For the core AI logic, orchestration, and machine learning.
- JavaScript Interop: Using the
javascript-asynclibrary to control a Mineflayer bot in Node.js. - Large Language Models: Leveraging Google Gemini for high-level reasoning and decision-making, orchestrated by Pydantic AI.
- Machine Learning: A custom "Instinct Engine" for reactive, low-level behaviors.
The agent operates autonomously within a Minecraft server, making decisions based on environmental awareness, chat messages, and in-game events. It can navigate, fight, communicate, and survive with a human-like behavioral protocol.
- 🤖 Autonomous Decision-Making: Powered by Google Gemini for context-aware reasoning and strategic planning, complemented by a persistent memory system.
- 👁️ Environmental Awareness: Scans the environment in real-time for entities, blocks, and players, while tracking events like attacks, deaths, and chat messages.
- ⚔️ Dynamic Combat System: Engages in both PvP and PvE combat automatically, featuring strategic retreat mechanics when health is low.
- 🎒 Intelligent Inventory Management: Capable of searching for items, managing equipment, dropping unneeded objects, and basic crafting.
- 🧭 Advanced Navigation: Utilizes Mineflayer's pathfinding to navigate to specific coordinates, track entities, and avoid obstacles.
- 🧠 Dual Memory System: Implements a long-term "Scratchpad" for persistent knowledge and a short-term event queue for immediate context.
- ⚡ Instinct Module: A custom-trained ML model that provides fast, instinctual responses to common scenarios, bypassing the need for LLM inference for simple actions.
The agent's architecture is designed for modularity, allowing clear separation between the high-level decision-making core and the low-level Minecraft interaction layer.
+------------------------------------+
| Minecraft Server (Java Edition) |
+------------------------------------+
|
+------------------------------------+
| Mineflayer Bot (Node.js) |
+------------------------------------+
| (via javascript-async)
+------------------------------------+
| Python Agent Core |
|------------------------------------|
| ├─ Agent Loop (Decision Cycle) |
| ├─ Toolsets |
| │ ├─ Chat |
| │ ├─ Combat |
| │ ├─ Movement |
| │ ├─ Inventory |
| │ ├─ Scan |
| │ └─ Scratchpad (Memory) |
| ├─ Listeners (Events, Messages) |
| ├─ Instinct Engine (ML Model) |
| └─ Plugins (PvP, AutoEat, etc.) |
+------------------------------------+
|
+------------------------------------+
| Pydantic AI + Google Gemini API |
+------------------------------------+
Follow these steps to get the agent running on your local machine.
- Python 3.12+
- Node.js 18+ with
npmoryarn - A running Minecraft Server (Java Edition) that you can connect to.
- Clone the repository:
git clone https://github.com/yourusername/minecraft-agent.git cd minecraft-agent
2Install Python dependencies:
This command installs the project in "editable" mode, which is great for development.
sh pip install -e .
3Set up environment variables:
Create a .env file from the example template.
sh cp .env.example .env
Then, edit the .env file with your credentials and server information.
All configuration is managed through the .env file.
# .env
# Required: Your Google Gemini API key
GOOGLE_API_KEY="your_google_api_key_here"
# Bot Configuration
BOT_NAME="Agent"
BOT_USERNAME="MinecraftUsername" # The username the bot will use to log in
# Server Connection
SERVER_HOST="127.0.0.1"
SERVER_PORT=25565To start the agent, run the main module:
python -m minecraft_agent.mainOr, if configured, use the command-line script:
agent_cliTo train the custom ML model for the Instinct Engine, run:
train_modelNote: Ensure you have training data available in the data/ directory.
The agent operates based on a set of core principles:
- Survival First: Prioritize safety, health, and resource gathering.
- Situational Awareness: Constantly monitor players, threats, and environmental changes.
- Pragmatism: Every action must serve a clear purpose toward a goal.
- Decisiveness: Act quickly and confidently based on available information.
- Social Interaction: Communicate with other players in Portuguese (pt-BR).
minecraft-agent/
├── data/ # Training data for the Instinct Model
├── src/minecraft_agent/
│ ├── agent/ # Core agent logic and decision loop
│ │ ├── agent.py
│ │ ├── agent_loop.py
│ │ └── agent_context.py
│ ├── tooling/ # Tools the agent can use
│ │ ├── chat/
│ │ ├── combat/
│ │ ├── inventory/
│ │ ├── movement.py
│ │ ├── scan/
│ │ └── scratchpad/
│ ├── events/ # Event handling logic
│ ├── instinct/ # The ML-based instinct engine
│ ├── plugins/ # Wrappers for Mineflayer plugins
│ │ ├── pathfinder.py
│ │ ├── pvp.py
│ │ ├── auto_eat.py
│ │ ├── armor_manager.py
│ │ └── blood_hound.py
│ ├── listeners/ # Listeners for bot events
│ ├── stubs/ # Type stubs for interop
│ └── main.py # Main entry point
├── pyproject.toml # Project metadata and dependencies
└── README.md
The agent is equipped with a variety of tools to interact with the world.
- Scan Environment: Gathers information about nearby entities, players, and blocks within a configurable radius.
- Send Message: Broadcasts messages to other players in the chat.
- Get Last Messages: Reads recent chat history to understand conversations.
- Engage Combat: Automatically attacks and pursues a target entity.
- Strategic Retreat: Disengages from combat and flees when health is critical.
- List Items: Lists all contents of the bot's inventory.
- Toss Item: Drops a specified item from inventory.
- Equip/Unequip: Manages armor and held items.
- Craft Item: Crafts a new item using a known recipe and available materials.
- Go to Position: Navigates to a specific set of coordinates.
- Go to Entity: Follows and approaches a target entity.
- Write to Scratchpad: Saves important information for long-term recall.
- Read from Scratchpad: Retrieves previously saved memories.
- Clear Scratchpad: Resets the agent's long-term memory.
This project was created for educational purposes as part of the Pupunha Code community. Contributions are welcome! Please feel free to fork the repository, make improvements, and submit a pull request.
- Fork the repository.
- Create your feature branch (
git checkout -b feature/AmazingFeature). - Commit your changes (
git commit -m 'Add some AmazingFeature'). - Push to the branch (
git push origin feature/AmazingFeature). - Open a Pull Request.
Distributed under the MIT License. See LICENSE for more information.
- Pupunha Code Community: For the opportunity to build and present this project.
- Pydantic AI: For the excellent and intuitive LLM framework.
- Mineflayer: For the comprehensive and powerful Minecraft bot API.
- Google Gemini: For powering the agent's cognitive abilities.
For questions or feedback, please open an issue on GitHub or reach out through the Pupunha Code community channels.
Built with ❤️ for Pupunha Code
Making AI play Minecraft, one block at a time.