In the previous post, I broke down the basic concepts of the Griptape AI framework, and now it’s time to put them into practice. We’ll try to use them to develop a small application that helps run a link-blog on Telegram.

The application will receive a URL, download its content, run it through an LLM to generate a summary, translate that summary into a couple of other languages, combine everything, and publish it to Telegram via a bot. The general flow can be seen in the diagram below:

What on Earth is Codex?

Good question, right? The thing is, until recently, OpenAI had a model called Codex, which was used as the foundation for autocompletion in GitHub Copilot. Then, OpenAI released a console agent for development, which they named, so no one would get confused, Codex. Everyone had a laugh at OpenAI’s naming skills , and life went on. Until the fateful day when a tweet like this appeared from Sam Altman:

• 1. Introduction: The Imperative of AI Crawler Management
• 2. Understanding robots.txt: The Foundation of Crawler Instruction
  • 2.1. Core Syntax and Directives
  • 2.2. File Placement and Formatting
  • 2.3. How Crawlers Interpret robots.txt
  • 2.4. Testing Your robots.txt
• 3. Identifying Key Crawler Types: AI Agents vs. Search Engine Bots
  • 3.1. Distinguishing Characteristics
  • 3.2. Categories of AI Crawlers and Their User Agents
    • 3.2.1. AI Crawlers for Model Training
    • 3.2.2. AI Crawlers for Live Retrieval and Search Assistance
  • 3.3. Standard Search Engine Crawlers (to be Allowed)
  • 3.4. Table of Prominent AI Crawler User Agents
• 4. Strategically Blocking AI Crawlers with robots.txt
  • 4.1. Targeting Specific AI User Agents
  • 4.2. Applying Rules to Specific Pages or Directories
  • 4.3. Ensuring Search Engines Are Not Blocked from Specific Pages
  • 4.4. The Challenge of “All Possible AI”
• 5. Advanced Methods for Granular AI Crawler Control
  • 5.1. HTML Meta Tags (Page-Level Control)
  • 5.2. HTTP X-Robots-Tag Headers (Server-Level Page Control)
  • 5.3. Server-Side Blocking
  • 5.4. Web Application Firewalls (WAFs) and Content Delivery Networks (CDNs)
  • 5.5. Table: Comparison of AI Crawler Control Mechanisms
• 6. Limitations and Best Practices
  • 6.1. robots.txt is a Directive, Not an Enforcement Mechanism
  • 6.2. Importance of Regular Review and Updates
  • 6.3. Testing robots.txt Changes
  • 6.4. Avoiding Common Pitfalls
  • 6.5. Log File Analysis
• 7. Conclusion: Implementing a Robust, Layered AI Crawler Defense

1. Introduction: The Imperative of AI Crawler Management

The proliferation of Artificial Intelligence (AI) has introduced a new class of web crawlers designed to gather vast quantities of data for training Large Language Models (LLMs) and powering AI-driven applications. While these advancements offer significant potential, website operators often require precise control over which content AI crawlers can access, particularly to protect intellectual property, sensitive information, or manage server resources. Simultaneously, maintaining visibility and crawlability for traditional search engine bots like Googlebot and Bingbot remains paramount for organic search performance.

Today we will look at Griptape, a framework for building AI applications, which offers a clean Pythonic API for those tired of LangChain’s abstraction layers. It provides primitives for building assistants, RAG systems, and integrating with external tools. Honestly, in my experience, most people tired of LangChain switch to custom-written wrappers around lower-level libraries like OpenAI or LiteLLM. But who knows, maybe they’re missing out. Let’s dive in.

A Bit of History

Personally, I’ve been hearing about Griptape for about a year and a half. As far as I remember, It started as a sort of LangChain competitor with quite similar primitives, but their paths gradually diverged. As of the time of the writing, it has 2.3k stars on GitHub, which is somewhat less than LangChain’s 109k, but still enough to consider the project quite mature. Besides the open-source framework, it has also developed its own cloud where you can run your applications, ETLs, and RAGs, and a visual builder, Griptape Nodes, allowing non-professionals to click together applications in minutes.

Alright, Here We Go

I got my hands on the Re:Camera from Seeed. Essentially, it’s a small box (a cube about 4 cm per side), wrapped in a heatsink. Inside, there’s a dual-core RISC-V based MPU (updated: only one core is visible to the system, the second one is apparently reserved for special operations), an ancient 8051 microcontroller, an OmniVision camera sensor, LEDs for illumination, Wi-Fi, BT, and, you know, all sorts of peripherals. RAM is a bit scarce, only 256 megabytes, so getting Greengrass on it will be problematic. You can connect Ethernet via a special dongle-adapter that barely stays put, but for development, there’s no point, because the camera shares its network over USB type C, and it’s easier to work that way. If you’re short on storage (and the device comes in 8 GB and 64 GB built-in storage options), you can stick in a MicroSD card. You can also stick the box to something metallic, as it has magnets on one side.