IndexNow-Automate your search engine URL submissions

IndexNow-Automate your search engine URL submissions

What is IndexNow?

IndexNow is an easy way for websites owners to instantly inform search engines and web crawlers used for information retrieval (“search engines”) about latest content changes on their website. In its simplest form, IndexNow is a simple ping so that search engines know that a URL and its content has been added, updated, or deleted, allowing search engines to quickly reflect this change in their search results.

Without IndexNow, it can take days to weeks for search engines to discover that the content has changed, as search engines don’t crawl every URL often. With IndexNow, search engines know immediately the “URLs that have changed, helping them prioritize crawl for these URLs and thereby limiting organic crawling to discover new content.”

IndexNow is offered under the terms of the Attribution-ShareAlike Creative Commons License and has support from Microsoft Bing, Naver, Seznam.cz, Yandex, Yep.

How IndexNow Works ?

IndexNow works in a very simple way. All you need to do is create a key on your server and then notify search engines that support IndexNow about the changes by sending a URL to the search engine. The steps are as follows:

1. Use the online key generation tool to create a key supported by the protocol.
2. Host the key in a text file named with the key value in the root directory of your website.
3. Once URLs are added, updated, or deleted, start sending the URLs. You can send a single URL or a batch of URLs per API request.
4. Sending a single URL is as simple as making a basic HTTP request that includes the changed URL and your key.

It works by using URLs like this:

https://api.indexnow.org/indexnow?url=url-changed&key=your-key

https://www.bing.com/indexnow?url=url-changed&key=your-key

https://searchadvisor.naver.com/indexnow?url=url-changed&key=your-key

https://search.seznam.cz/indexnow?url=url-changed&key=your-key

https://yandex.com/indexnow?url=url-changed&key=your-key

https://indexnow.yep.com/indexnow?url=url-changed&key=your-key

Sample Request

POST /IndexNow HTTP/1.1
Content-Type: application/json; charset=utf-8
Host: api.indexnow.org
{
  "host": "www.example.org",
  "key": "16ed3fbe0dff457c9bb3cc71966a8290",
  "keyLocation": "https://www.example.org/16ed3fbe0dff457c9bb3cc71966a8290.txt",
  "urlList": [
      "https://www.example.org/url1",
      "https://www.example.org/folder/url2",
      "https://www.example.org/url3"
      ]
}

Is Google use IndexNow ?

As of now, Google does not officially support the IndexNow protocol. IndexNow is primarily adopted by Bing and Yandex. Google has its own methods of crawling and indexing web pages, including using sitemaps and its search console tools. Google has not integrated IndexNow into its search engine as part of its official indexing process.

However, it’s always possible that Google could consider integrating IndexNow or similar protocols in the future, given the growing interest in real-time indexing and automated updates to search engines.

Automate Sitemap Url Application ?

If your applications are on a small scale, you might not need to use the IndexNow feature. According to Sitemap.org, it can be stored in a standard XML file with up to 50,000 URLs and a maximum file size of 50MB. For projects like ours, which have million links and are constantly growing, it is essential to use this method. For Google, the standard sitemap structure should still be preferred.

For that reason, I created an application to synchronize sitemaps with search engines.

Database Setup

First at all you must create database and sitemap data.

CREATE database examplecomdb_sitemap;

CREATE TABLE `tbindexnow` (
  `indexnowid` bigint(20) NOT NULL AUTO_INCREMENT,
  `sitemapid` bigint(20) DEFAULT NULL,
  `lastupdatedate` timestamp NULL DEFAULT current_timestamp(),
  PRIMARY KEY (`indexnowid`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4;

CREATE TABLE `tbsitemap` (
  `sitemapid` bigint(20) NOT NULL AUTO_INCREMENT,
  `creationdate` timestamp NULL DEFAULT current_timestamp(),
  `loc` varchar(2000) NOT NULL,
  `lastmod` varchar(100) DEFAULT NULL,
  `changefreq` varchar(50) DEFAULT NULL,
  `priority` double DEFAULT 0.5,
  `sitemapname` varchar(100) DEFAULT NULL,
  `relatedproductid` varchar(100) DEFAULT '0',
  PRIMARY KEY (`sitemapid`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 ;

tbindexnow Table Documentation

tbindexnow is the current reading sitemap detail store. First application use insert empty a record like this.

tbsitemap Table Documentation

Overview

The tbsitemap table is used to store all URLs that are included in sitemaps submitted to search engines. It supports grouping URLs into different sitemaps using the sitemapname field, allowing better organization and management.

A sitemap file can contain a maximum of 50,000 URLs, so sitemapname helps differentiate and extract specific sitemap groups.

Usage Scenarios

1. Storing URLs for Sitemap Generation:

Each row represents a URL that should be included in an XML sitemap.

2. Grouping URLs by Sitemap:

• The sitemapname column is used to define different sitemap files.
• Example: If a sitemap can contain a maximum of 50,000 URLs, sitemapname helps extract and generate separate sitemap files.

3. Providing Metadata for Search Engines:

• lastmod, changefreq, and priority provide additional details for search engines to optimize indexing.

Application Part

This C# program processes sitemaps from a MySQL database and submits them to search engines using the IndexNow protocol. It follows a modular architecture with separate classes for configuration loading, database operations, API requests, and logging.

Key components:

• ConfigLoader: Reads and deserializes the configuration file.
• SitemapProcessor: Manages sitemap retrieval and submission flow.
• DatabaseHelper: Handles database queries for fetching and updating sitemap IDs.
• ApiExecutor: Sends sitemap URLs to search engines via HTTP requests.
• LogHelper: Logs errors to a file for debugging.

The program ensures efficient sitemap processing using a loop-based approach with configurable pagination and sleep intervals to avoid overloading services. 🚀

You can access the source.

Configuration Setup

The application reads configuration settings from a JSON file. Modify the parameters based on your domain and setup:

example.com.json

{
  "ConnectionString": "Server=localhost;Database=examplecomdb_sitemap;User Id=username;Password=password;",
  "PageSize": 1000,
  "IndexNowId": 1,
  "Host": "www.example.com",
  "ApiKey": "24c4d7a83f1d0e7617511360e761751136",
  "KeyLocation": "https://www.example.com/24c4d7a83f1d0e7617511360e761751136.txt",
  "SleepTime": 10000,
  "IndexNowEndpoints": {
    "Yandex": "https://www.yandex.com/IndexNow",
    "IndexNow": "https://api.indexnow.org/IndexNow",
    "Bing": "https://www.bing.com/indexnow"
  }
}

Configuration Parameters

• ConnectionString: Database connection settings.
• PageSize: Number of records to process per request.
• IndexNowId: Identifier for the indexing process.
• Host: Website domain.
• ApiKey: Unique key for IndexNow authentication.
• KeyLocation: URL where the IndexNow key is stored.
• SleepTime: Delay in milliseconds between requests (10,000 ms = 10 seconds).
• IndexNowEndpoints: API endpoints for Yandex, IndexNow, and Bing submissions.

Running the Application

Run the application using the following command:

SitemapIndexNowApp.exe example.com.json

This will process sitemaps from the database and submit them to the search engines using the configured IndexNow endpoints.

Making a conclusion

👨‍👦‍👦 Leave a comment, I am free for discussion with your any kind technical question.