Optimizing the optimizer: How Botify is supercharging SEO with AI

Back in 1994, two Stanford students — no, not those two — put together an index of the web’s most interesting sites. When Jerry Wang and David Filo published the first iteration of Yahoo, it numbered around 600 webpages. Today, roughly that many new web pages are published every minute-and-45-seconds.

The challenges of discovery, though, remain largely the same then and now. 

Whether it was manually submitting your site for inclusion, as in Yahoo’s early days, or globe-spanning search engines and social networks doing the surfacing, getting your site noticed has never been easy or straightforward (at least since Wang and Filo stopped accepting personal submissions).

From those humble beginnings sprung a whole industry — search engine optimization — dedicated to helping businesses improve their search results and gain recognition. One such company, Botify, was born in 2012 with the goal of helping the largest and most complex websites gain more traffic and improve organic search results. 

“The internet is growing larger and larger by every second,” Robert Rothschild, chief marketing officer at Botify, said during an interview with the Transformation Debrief podcast. “The amount of content that is being created every moment of every day is massive and it's exponential. And the ability for all of the search engines, Google included, to keep up with that information is incredibly challenging.” 

And it’s a major challenge for those looking to be searched for, too, one they must solve if they hope to be found and win over customers. According to one study, the first three results on search garner about 55% of the total clicks. And only about 0.78% of people bother to click on the second page of search results, according to Backlinko.

To deliver for its customers, Botify continues to reinvent how companies perform SEO. From those early days of the manual indexing of websites, to the eras of keyword stuffing and anchor links, Botify is now helping to change the entire industry yet again by applying machine learning and artificial intelligence to the job of improving SEO.

“The biggest transformation that I'm seeing in the industry is the addition of the overlay of artificial intelligence and machine learning on top of the data,” Rothschild said. “The next piece of that is when you layer on automation, which is about to introduce the capability to implement changes automatically, without relying upon human intervention. And that will free up SEOs to focus on much more strategic activities.”

Wrestling the immense content industry

In the early ‘90s, the content industry did not exist. Sure, there was entertainment and news, but the term “content” — the mass of digital media and data that has flooded the world — was not yet a recognized entry into the lexicon. (It would take until about 1997 for “content” to be recognized as we now know it.) It took the adoption of the World Wide Web, then social media, cloud computing, smartphones, and all varieties of digital media to create the contemporary content industry.

According to the Internet Live Stats counter, there are currently about 1.985 billion websites in the world. And just about every one of them is looking for its own little slice of attention. Botify scans many of these websites, with more than 4.3 billion web pages crawled and analyzed, consisting of the largest corporate websites with sometimes hundreds of thousands, or even millions, of pages. The company also analyzes 61 billion search engine requests monthly, and more than 900 million real Google search queries from over 150 countries. 

Botify stores approximately six petabytes of data, an enormous amount for any company. On a monthly basis, the company will parse five petabytes of that data. Botify handles the load with tight integration with Google Cloud’s BigQuery data warehouse. It also utilizes Google Compute Engine and BigTable to help crawl client websites, as well as Looker’s analytics tools to help visualize and share data.  

Even the largest of companies with the most resources to grapple with SEO issues struggle on a daily basis. The modern web is a gargantuan and complicated ecosystem. According to Botify, most corporate web pages are never actually crawled by a search engine.

“Turns out that, when you actually look at the figures, you will discover that the largest websites get 25 to 30 percent coverage,” Pierre Couzy, chief technology officer at Botify, said. “Meaning that 70 percent of the website has not been visited before by a search engine.”

One of Botify’s prime objectives when helping a company with its SEO operations is to increase the search coverage of its pages. But improving search performance is only another data point without the ability to act upon the new knowledge.

“When we try to automate, we focus on the biggest gain,” Couzy said. “What's the impact of a specific change? Those two KPIs drive how we chat internally about the product, because it gives us a measure of success for the customer. Being more visible means more landing clicks, because you will have better search results. And the second one is, how big is the improvement?”

Artificial intelligence is the new search differentiator

For most of the last two decades, the way to gain an edge over older and less nimble competitors was to have a robust and intuitive digital presence. We still talk about digital transformation on a daily basis, even if the ultimate meaning of what constitutes digital transformation changes on a quarterly basis.

In 2022, just being digital is not the differentiator it used to be. 

“I was attending a symposium led by Gartner,” Rothschild said. “And one of the analysts mentioned that digital is no longer a differentiator. And I thought that was a tremendously provocative statement. And the reason for it is that every organization is digital first now. That's where you start. It's table stakes at this stage.”

Instead, the new differentiator in the marketplace is the ability to deftly handle complexity and derive timely insights from massive datasets. Artificial intelligence and machine learning have made this possible and within easy reach for most companies. Botify is able to use machine learning and automation to crawl its clients’ websites and come up with actionable insights quickly, such as how to speed up the loading of web pages or improve keyword placement. 

“Being able to implement a change rapidly, see what the impact is, and then leverage AI and machine learning to optimize that performance over time, is a tremendous advantage for marketers in every industry,” Rothschild said.

If you’d like to learn more about Botify's technology journey, we have two more exciting channels for you. You can watch a video conversation or listen to an extended podcast with Botify's Pierre Couzy, chief technology officer, and Robert Rothschild, chief marketing officer. In each segment, they go even more in depth on the state of digital discovery and the myriad ways artificial intelligence and machine learning continue to evolve their business and others.