Navigating Bot Traffic: Insights and Strategies from WP Engine DE{CODE}
Follow us
Bots now account for over 50% of internet traffic, forcing agencies and site owners to adapt their strategies around intelligent traffic management and bot visibility. To explore this change, WP Engine hosted a panel at its 2026 DE{CODE} conference that featured both internal experts and external industry leaders.
WP Engine Senior Product Manager Jenna Menue moderated the session titled Good Bots, Bad Bots: How to Win Bot Management. Joined by Cloudflare Director of Product Brian Becker, Useful Group CTO PeterJohn Hunt, and WP Engine Senior Product Manager Krystal O'Connor, the panel provided valuable insights into the rise of complex bots and the future of intelligent traffic management.
Gray Bots: To Block or Not to Block?
A central theme is the growing complexity of bot traffic. O’Connor set the stage by highlighting the types of bots we’re seeing online. Everyone knows there are “good bots” like search engine crawlers and “bad bots” with malicious intentions. With the recent rise of AI crawlers and automated agents, a cryptic third category has started to surge. These “gray bots” are now arriving on sites with ambiguous intentions.
Hunt echoed her framing, saying, "It feels like five years of traffic has been condensed into the last six months... Patterns everybody took for granted allowed us to operate on assumptions. Now, everybody has to worry about a lot, because you want good traffic, but there’s just a lot more traffic to deal with.”
For agencies and site owners, this presents a unique challenge: how do you identify, manage, and mitigate unknown traffic without disrupting real user experiences?
Visibility and Logging: The Foundation of Bot Management
The first step in effective bot management is visibility. Without a clear understanding of a site’s traffic, it’s impossible to build a nuanced strategy.
“If you carve out the path for the good bots and you carve out the path for the bad bots, we’re just left talking about the gray area,” said Becker.
Strategic logging allows operators to adopt a cautious approach to bots without blocking every unverified request. Users can start by observing and logging patterns to watch how incoming traffic behaves before taking action to ensure they don’t accidentally block legitimate traffic.
“There are a lot of reasons why we want to block bots, but I think you can bucket most of those reasons into either it damages your site, or it results in fraud and abuse. If you could prevent those two things from happening, there's a different question to ask,” Becker continued. “If these bots are buying things and they're good purchases, logging this behavior could help you determine that and make a decision. Maybe you don't want to block it.”
Striking the Right Balance: Bot Mitigation Best Practices
During the session, Becker presented a few practical strategies for balancing malicious bot mitigation with genuine visitor experiences:
- CAPTCHA: Using CAPTCHA instead of hard blocks can reduce suspicious activity, but may create additional friction for legitimate users.
- Event-Based Adjustments: Temporarily loosening restrictions during expected high-traffic periods allows site operators to accommodate legitimate users who may initially appear unverified.
- Bot Incentives: Providing clear API guidelines encourages helpful bots, like AEO and GEO crawlers, to self-identify.
Hunt advised agencies to contextualize traffic management for each client. "You now have to set your strategy related to your client’s industry. I have to know not just the specific traffic, but the specific traffic in the context of the client, because there will be different patterns of what good, bad, and neutral are."
Taking an iterative approach is key. Mitigation strategies can and should be adjusted as you learn more about specific traffic patterns.
WP Engine’s New Features for Enhanced Control
WP Engine will soon roll out a few new features aimed at simplifying bot management:
- Traffic Visibility Dashboard: Global Edge Security customers will be able to filter, view, and challenge traffic by agent category, region, and behavior data.
- Edge Full Page Caching: WP Engine will expand its edge caching capabilities across its Advanced Network and GES.
As O’Connor explained, "[Edge Full Page Cache] provides full-page HTML caching at the edge. This is obviously good for human users because they'll get your content faster, but it’s also good for solving the bots problem because all that traffic isn't bogging down your server.”
These tools offer deeper insight and actionable control over how traffic impacts performance.
Practical Wisdom for Beginners
For those new to bot management, the panelists emphasized starting simple.
“The entire internet community is dealing with this at the same time, so we all need to learn from each other,” said O’Connor. “Use trial and error to see what's going to work for your site, for your site persona, for your customer.”
Leveraging visualization tools can reduce feelings of being overwhelmed by illuminating the unknowns.
“Visualization helps reduce a lot of the stress because it's no longer an unknown, it's a ‘now what do we do with this information?’” said Hunt. “Whatever your tool is for visualization can help explain [bot traffic patterns] to your customer, so now you've got more time to be able to figure out what we're fixing, how we're gonna fix it, and what's next.”
Collaboration with other teams or agencies can further streamline the implementation of a bot management strategy.
“It's great to have these healthy discussions about how we're building strategy, and it does not start or end here,” said Menue. “I'd encourage you to chat with others in your network about what they're doing for bots. Keep sharing your knowledge, as we all navigate this together.”
Bot Management is the Next Non-Negotiable
Bots are getting more sophisticated, and internet traffic patterns are murkier than ever, so bot management is all about observation and small, incremental changes.
Now is the time to start integrating visibility tools and collaborating with peers and experts. To hear more, visit WP Engine’s events page to watch all sessions from DE{CODE} 2026 on demand.