Manticore Search Engine-Fast database for search
Manticore Search Engine-Fast database for search
Introduction
Manticore Search is a high-performance, multi-storage database designed for search and analytics. It offers ultra-fast full-text search, real-time indexing, and advanced capabilities like vector search and columnar storage for efficient data analysis. Engineered for scalability, it seamlessly handles both small and large datasets, providing powerful insights for modern applications.
Originally launched in 2017 as an evolution of the Sphinx Search engine, Manticore Search builds on its predecessor’s strengths while introducing significant improvements. As an open-source project available on GitHub, it has undergone a near-complete rewrite, fixing hundreds of bugs and enhancing functionality, as detailed in its Changelog. The result is a modern, fast, and lightweight database with exceptional full-text search capabilities.
Why Should Choose Manticore Search Engine ?
Manticore Search offers support for both row-wise storage (similar to MySQL and Postgres) and columnar storage (similar to Redshift, Clickhouse, BigQuery, and Druid), making it incredibly fast for small data sets that can fit in RAM and still maintaining speed when your data is significantly larger than the RAM size.
Manticore Search is: 15x faster than Elasticsearch on small data, 4x faster on big data, and 29x faster for log analytics.
Manticore Search, initially designed as a pure full-text search engine, boasts exceptional full-text capabilities: with over 20 full-text operators and more than 20 ranking factors,
Manticore support 3 different way,
- port 9306 for MySQL clients
- port 9308 for HTTP/HTTPS connections
- port 9312 for HTTP/HTTPS, and connections from other Manticore nodes and clients based on Manticore binary API
Manticore Search seamlessly supports HTTP requests, making it easy to integrate with any HTTP client. This flexibility allows you to effortlessly adapt it to your codebase, enabling smooth and efficient development.
Sample usage
POST 127.0.0.1:9308/search
-d
'{
"table" : "tbcustomer",
"query":
{
"match":
{
"*" : "find joe"
}
},
"_source": ["name","familyname"],
"limit": 1
}'
Manticore Setup
Before start the document you should already install mariadb/mysql and manticore. In this document you must install all package.
Default Manticore Setting after install Database this setting will be update.
root@desktop-ao725:~# less /etc/manticoresearch/manticore.conf;
searchd {
listen = 127.0.0.1:9312
listen = 127.0.0.1:9306:mysql
listen = 127.0.0.1:9308:http
log = /var/log/manticore/searchd.log
query_log = /var/log/manticore/query.log
pid_file = /var/run/manticore/searchd.pid
data_dir = /var/lib/manticore
}
/etc/manticoresearch/manticore.conf (END)
For this document I was created Sample Database
CREATE DATABASE manticore_sample;
-- manticore_sample.tbcustomer definition
CREATE TABLE `tbcustomer` (
`customerid` int(11) NOT NULL AUTO_INCREMENT,
`firstname` varchar(50) NOT NULL,
`lastname` varchar(50) NOT NULL,
`email` varchar(100) DEFAULT NULL,
`phone` varchar(20) DEFAULT NULL,
`address` text DEFAULT NULL,
`city` varchar(50) DEFAULT NULL,
`country` varchar(50) DEFAULT NULL,
`postalcode` varchar(20) DEFAULT NULL,
`creationdate` timestamp NULL DEFAULT current_timestamp(),
PRIMARY KEY (`customerid`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_uca1400_ai_ci;
-- manticore_sample.tbcustomer random data insertion
CREATE PROCEDURE `manticore_sample`.`InsertRandomCustomers`()
BEGIN
DECLARE i INT DEFAULT 1;
DECLARE firstnames VARCHAR(500) DEFAULT 'John,Michael,David,James,Robert,William,Joseph,Charles,Thomas,Daniel,Matthew,Anthony,Mark,Donald,Paul,Steven,Andrew,Kenneth,Joshua,Kevin';
DECLARE lastnames VARCHAR(500) DEFAULT 'Smith,Johnson,Williams,Brown,Jones,Miller,Davis,Garcia,Rodriguez,Wilson,Martinez,Anderson,Taylor,Thomas,Hernandez,Moore,Martin,Jackson,Thompson,White';
DECLARE cities VARCHAR(500) DEFAULT 'New York,Los Angeles,Chicago,Houston,Phoenix,Philadelphia,San Antonio,San Diego,Dallas,San Jose';
WHILE i <= 10000 DO
INSERT INTO tbcustomer (firstname, lastname, email, phone, address, city, country, postalcode)
VALUES (
ELT(FLOOR(1 + (RAND() * 20)), 'John', 'Michael', 'David', 'James', 'Robert', 'William', 'Joseph', 'Charles', 'Thomas', 'Daniel', 'Matthew', 'Anthony', 'Mark', 'Donald', 'Paul', 'Steven', 'Andrew', 'Kenneth', 'Joshua', 'Kevin'),
ELT(FLOOR(1 + (RAND() * 20)), 'Smith', 'Johnson', 'Williams', 'Brown', 'Jones', 'Miller', 'Davis', 'Garcia', 'Rodriguez', 'Wilson', 'Martinez', 'Anderson', 'Taylor', 'Thomas', 'Hernandez', 'Moore', 'Martin', 'Jackson', 'Thompson', 'White'),
CONCAT(LOWER(ELT(FLOOR(1 + (RAND() * 20)), 'john', 'michael', 'david', 'james', 'robert', 'william', 'joseph', 'charles', 'thomas', 'daniel', 'matthew', 'anthony', 'mark', 'donald', 'paul', 'steven', 'andrew', 'kenneth', 'joshua', 'kevin')), '.',
LOWER(ELT(FLOOR(1 + (RAND() * 20)), 'smith', 'johnson', 'williams', 'brown', 'jones', 'miller', 'davis', 'garcia', 'rodriguez', 'wilson', 'martinez', 'anderson', 'taylor', 'thomas', 'hernandez', 'moore', 'martin', 'jackson', 'thompson', 'white')),
FLOOR(RAND() * 1000), '@example.com'),
CONCAT('+90', FLOOR(RAND() * 900000000 + 100000000)),
CONCAT(FLOOR(RAND() * 999) + 1, ' Main Street'),
ELT(FLOOR(1 + (RAND() * 10)), 'New York', 'Los Angeles', 'Chicago', 'Houston', 'Phoenix', 'Philadelphia', 'San Antonio', 'San Diego', 'Dallas', 'San Jose'),
'USA',
CONCAT(FLOOR(RAND() * 90000 + 10000))
);
SET i = i + 1;
END WHILE;
END
-- Execute Store Procedure
{ CALL manticore_sample.InsertRandomCustomers() }
After generate Sample Data we must update
mantisearch.conf setting with your own server setting.
if you want to access remote desktop you must
change
listen = 127.0.0.1:9308:http to listen = 0.0.0.0:9308:http
/etc/manticoresearch/manticore.conf
# Manticore Search Configuration File
## Data Source Configuration
source tbcustomer_source
{
type = mysql
sql_host = 127.0.0.1
sql_user = username
sql_pass = password
sql_db = manticore_sample
sql_port = 3306
sql_query = SELECT customerid, firstname, lastname, email, phone, creationdate FROM tbcustomer
sql_attr_uint = customerid
sql_field_string = firstname
sql_field_string = lastname
sql_field_string = email
sql_field_string = phone
sql_field_string = creationdate
}
## Index Definition
index tbcustomer
{
source = tbcustomer_source
path = /var/lib/manticore/tbcustomer
}
## Search Daemon Configuration
searchd
{
listen = 127.0.0.1:9312
listen = 127.0.0.1:9306:mysql
listen = 0.0.0.0:9308:http
log = /var/log/manticore/searchd.log
query_log = /var/log/manticore/query.log
pid_file = /var/run/manticore/searchd.pid
}
After modifying the configuration settings, restart the Manticore service. Start indexer service and change file owner status.
root@desktop-ao725:~# root@desktop-ao725:~# systemctl restart manticore
root@desktop-ao725:~# indexer --config /etc/manticoresearch/manticore.conf tbcustomer --rotate
Manticore 7.0.0 92c650401@25013002
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)
using config file '/etc/manticoresearch/manticore.conf'...
indexing table 'tbcustomer'...
collected 11000 docs, 0.7 MB
creating lookup: 11.0 Kdocs, 100.0% done
sorted 0.1 Mhits, 100.0% done
total 11000 docs, 787463 bytes
total 1.551 sec, 507504 bytes/sec, 7089.28 docs/sec
total 3 reads, 0.005 sec, 530.3 kb/call avg, 1.8 msec/call avg
total 20 writes, 0.020 sec, 208.2 kb/call avg, 1.0 msec/call avg
rotating tables: successfully sent SIGHUP to searchd (pid=20698).
root@desktop-ao725:~# chown -R manticore:manticore /var/lib/manticore
root@desktop-ao725:~# chmod -R 755 /var/lib/manticore
root@desktop-ao725:/var/lib/manticore# systemctl restart manticore
After indexer finish check the tbcustomer table from manticore engine is exist.
root@desktop-ao725:~# mysql -h0 -P9306
WARNING: option --ssl-verify-server-cert is disabled, because of an insecure passwordless login.
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 7.0.0 92c650401@25013002 git branch manticore-7.0.0...origin/manticore-7.0.0
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Support MariaDB developers by giving a star at https://github.com/MariaDB/server
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> show databases;
+-----------+
| Databases |
+-----------+
| Manticore |
+-----------+
1 row in set (0,001 sec)
MySQL [(none)]> use Manticore;
Database changed
MySQL [Manticore]> show Tables;
+------------+-------+
| Table | Type |
+------------+-------+
| tbcustomer | local |
+------------+-------+
1 row in set (0,001 sec)
MySQL [Manticore]> select * from tbcustomer;
+------+-----------+----------+---------------------------------+--------------+---------------------+
| id | firstname | lastname | email | phone | creationdate |
+------+-----------+----------+---------------------------------+--------------+---------------------+
| 1 | Paul | Williams | [email protected] | +90671293181 | 2025-02-13 16:53:28 |
| 2 | Joshua | Johnson | [email protected] | +90614237478 | 2025-02-13 16:53:28 |
| 3 | Steven | Anderson | [email protected] | +90380113068 | 2025-02-13 16:53:28 |
| 4 | William | Brown | [email protected] | +90135708526 | 2025-02-13 16:53:28 |
| 5 | Charles | Brown | [email protected] | +90122012831 | 2025-02-13 16:53:28 |
| 6 | Joshua | Davis | [email protected] | +90489134906 | 2025-02-13 16:53:28 |
| 7 | Donald | Martinez | [email protected] | +90412573861 | 2025-02-13 16:53:28 |
| 8 | Mark | Jones | [email protected] | +90909515773 | 2025-02-13 16:53:28 |
| 9 | Daniel | Jackson | [email protected] | +90135950437 | 2025-02-13 16:53:28 |
| 10 | Mark | Wilson | [email protected] | +90879398405 | 2025-02-13 16:53:28 |
| 11 | Mark | Wilson | [email protected] | +90282762774 | 2025-02-13 16:53:28 |
| 12 | Joshua | Williams | [email protected] | +90452409818 | 2025-02-13 16:53:28 |
| 13 | William | Smith | [email protected] | +90705214579 | 2025-02-13 16:53:28 |
| 14 | Michael | White | [email protected] | +90205350861 | 2025-02-13 16:53:28 |
| 15 | Michael | Garcia | [email protected] | +90442190342 | 2025-02-13 16:53:28 |
| 16 | James | Garcia | [email protected] | +90554299511 | 2025-02-13 16:53:28 |
| 17 | Michael | Smith | [email protected] | +90370187828 | 2025-02-13 16:53:28 |
| 18 | Steven | Martinez | [email protected] | +90987196628 | 2025-02-13 16:53:28 |
| 19 | Donald | Garcia | [email protected] | +90546122744 | 2025-02-13 16:53:28 |
| 20 | Anthony | Thompson | [email protected] | +90292202985 | 2025-02-13 16:53:28 |
+------+-----------+----------+---------------------------------+--------------+---------------------+
20 rows in set (0,010 sec)
MySQL [Manticore]>
We can access 2 different way one of them native Mysql native connection second one via http.
Http Sample
Request
POST /search HTTP/1.1
Content-Length: 118
Content-Type: application/json
Host: desktop-ao725:9308
User-Agent: HTTPie
{
"table" : "tbcustomer",
"query":
{
"match":
{
"*" : "John"
}
}
}
Response
{
"took": 8,
"timed_out": false,
"hits": {
"total": 1042,
"total_relation": "eq",
"hits": [
{
"_id": 466,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Taylor",
"email": "[email protected]",
"phone": "+90541030792",
"creationdate": "2025-02-13 16:53:29"
}
},
{
"_id": 901,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Martin",
"email": "[email protected]",
"phone": "+90168596903",
"creationdate": "2025-02-13 16:53:30"
}
},
{
"_id": 1141,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Rodriguez",
"email": "[email protected]",
"phone": "+90783170469",
"creationdate": "2025-02-13 17:15:49"
}
},
{
"_id": 1211,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Jones",
"email": "[email protected]",
"phone": "+90606020728",
"creationdate": "2025-02-13 17:15:49"
}
},
{
"_id": 1797,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Jones",
"email": "[email protected]",
"phone": "+90381190256",
"creationdate": "2025-02-13 17:15:51"
}
},
{
"_id": 2009,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Jones",
"email": "[email protected]",
"phone": "+90825966940",
"creationdate": "2025-02-13 17:15:51"
}
},
{
"_id": 2120,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Taylor",
"email": "[email protected]",
"phone": "+90221217591",
"creationdate": "2025-02-13 17:15:51"
}
},
{
"_id": 3082,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Taylor",
"email": "[email protected]",
"phone": "+90795463240",
"creationdate": "2025-02-13 17:15:53"
}
},
{
"_id": 3954,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Rodriguez",
"email": "[email protected]",
"phone": "+90593916762",
"creationdate": "2025-02-13 17:15:55"
}
},
{
"_id": 3998,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Rodriguez",
"email": "[email protected]",
"phone": "+90599466513",
"creationdate": "2025-02-13 17:15:55"
}
},
{
"_id": 4250,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Rodriguez",
"email": "[email protected]",
"phone": "+90681918427",
"creationdate": "2025-02-13 17:15:55"
}
},
{
"_id": 4293,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Smith",
"email": "[email protected]",
"phone": "+90997585601",
"creationdate": "2025-02-13 17:15:56"
}
},
{
"_id": 4561,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Smith",
"email": "[email protected]",
"phone": "+90236734326",
"creationdate": "2025-02-13 17:15:56"
}
},
{
"_id": 4729,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Martin",
"email": "[email protected]",
"phone": "+90355355205",
"creationdate": "2025-02-13 17:15:56"
}
},
{
"_id": 5703,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Taylor",
"email": "[email protected]",
"phone": "+90386470621",
"creationdate": "2025-02-13 17:15:58"
}
},
{
"_id": 6597,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Martin",
"email": "[email protected]",
"phone": "+90208468622",
"creationdate": "2025-02-13 17:16:00"
}
},
{
"_id": 6654,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Taylor",
"email": "[email protected]",
"phone": "+90120357132",
"creationdate": "2025-02-13 17:16:00"
}
},
{
"_id": 6938,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Smith",
"email": "[email protected]",
"phone": "+90462542648",
"creationdate": "2025-02-13 17:16:01"
}
},
{
"_id": 7366,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Jones",
"email": "[email protected]",
"phone": "+90207082582",
"creationdate": "2025-02-13 17:16:02"
}
},
{
"_id": 8831,
"_score": 2575,
"_source": {
"firstname": "John",
"lastname": "Jones",
"email": "[email protected]",
"phone": "+90746512935",
"creationdate": "2025-02-13 17:16:04"
}
}
]
}
}
Manticore json struct below
{
"took": Elapsed Time,
"timed_out": false,
"hits": {
"total": , Founded Row count
"total_relation": "eq", This means say equal
"hits": [
{
"_id": source id,
"_score": ,
"_source": {
manticore.conf source fields
}
}
]
}
}
Manticore Sample Usage
Let’s connect to Manticore using the MySQL client:
mysql -P9306 -h0
The most simple full-text query in Manticore is an enumeration of words. The SQL statement used for definining a full-text match is ‘MATCH()’, which has a single parameter — the query string.
By comparison, in MySQL, for example, the syntax is ‘MATCH(list_of_columns) AGAINST(query_string)’:
SELECT * FROM tbcustomer WHERE MATCH('name,familyname') AGAINST('john doe');
The implicit operator is ‘AND’ and, by default, the words will be searched in all full-text fields available:
SELECT * FROM tbcustomer WHERE MATCH('john martin');
For boolean searches the OR ( ‘|’) can be used:
select * from tbcustomer where match('john martin | anderson');
The OR operator has higher precedence than AND, so the query ‘find me fast|slow’ is translated as ‘find me (fast|slow)’:
select * from tbcustomer where match('john | anderson');
For negations, operator NOT can be specified with ‘-’ or ‘!’ :
select * from tbcustomer where match('john !anderson -silva');
It must be noted that full negation queries are not supported and it’s not possible to run just ‘!fast’.
Another base operator is MAYBE. The term defined by MAYBE can be present or not in the documents. If it’s present, it will influence the ranking, and documents having it will be ranked higher
select * from tbcustomer where match('john MAYBE anderson');
Basic operators — fields
If we want to limit to search to a field, the operator ‘@’ can be used:
select * from tbcustomer where match('@email steven');
We can also specify multiple fields to limit the search:
select * from tbcustomer where match('@(firstname,email) steven');
Fuzzy search
Fuzzy matching allows us to match only some of the words from the query string
select * from tbcustomer where match('"fox bird lazy dog"/3');
In this case we specify that it’s fine to match only 3 of the words. A fuzzy of ‘/1’ is equivalent of an OR boolean search, which fuzzy of ‘/N’, where N is the number of input words, is equivalent of an AND search.
Instead of specifying number of words, we can use a fraction between 0.0 and 1.0. This can be helpful when dealing with queries of varying lengths. For example:
select * from tbcustomer where match('"fox bird lazy dog"/0.75');
This query would match any document that has at least three out of the four words ‘fox’, ‘bird’, ‘lazy’, and ‘dog’.
Manticore cluster
Manticore Search is a highly distributed system that provides all the necessary components to create a highly available and scalable database for search. This includes:
- distributed table for sharding
- Mirroring for high availability
- Load balancing for scalability
- Replication for data safety
Manticore Search offers great flexibility in terms of how you set up your cluster. There are no limitations, so it’s up to you to design your cluster according to your needs. Simply learn about the tools mentioned above and use them to achieve your desired goal.
for more detail https://manual.manticoresearch.com/Creating_a_cluster/Creating_a_cluster
Sample Demo Application
I was created sample demo application communicate Manticore via Http.
A simple PHP client application to interact with Manticore Search to perform search queries on customer data indexed…github.com
The ManticoreDemoClientApp is likely a client application that interacts with Manticore Search, a full-text search engine. This application would provide a demo interface for searching data indexed in Manticore, allowing users to execute queries, retrieve results, and display them in a user-friendly format.
Key features of such an app might include:
- Search Interface: A UI for users to input search queries.
- Query Execution: Connecting to a Manticore Search server to execute queries.
- Displaying Results: Showing search results in a structured or formatted manner, possibly with pagination.
- Error Handling: Handling search errors (e.g., no results, query issues).
- Integration with Manticore: Using Manticore Search APIs to fetch data.
The app might be used as a demonstration of how Manticore can be integrated into an application for efficient, high-performance search use cases. If you pull this project from GitHub, you can likely expect an example app showcasing how to interact with Manticore Search in a real-world scenario.
Main manticore logic below full text search and get by id usage exist.
// https://github.com/iatasoy/ManticoreDemoClientApp/blob/main/src/manticore/ManticoreSearchClient.php
<?php
use GuzzleHttp\Client;
use GuzzleHttp\Exception\RequestException;
class ManticoreSearchClient
{
private Client $client;
public function __construct(string $apiUrl, int $timeout = 10)
{
$this->client = new Client([
'base_uri' => $apiUrl,
'timeout' => $timeout,
]);
}
public function search(string $index, string $query, array $fieldList, int $size): array
{
try {
// Replace `+` or `%20` with `&` to match both terms in Manticore
$query = str_replace(['+', '%20'], '|', $query);
$response = $this->client->post('/search', [
'json' => [
'index' => $index,
'query' => [
'match' => [
'*' => $query
]
],
'fields' => $fieldList,
'size' => $size,
'from' => 0
],
]);
return json_decode($response->getBody(), true);
} catch (RequestException $ex) {
return ['error' => $ex->getMessage()];
}
}
public function searchById(string $index, int $id): array
{
try {
$response = $this->client->post('/search', [
'json' => [
'index' => $index,
'query' => [
'bool' => [
'must' => [
['equals' => ['id' => $id]]
]
]
],
'size' => 1
],
]);
return json_decode($response->getBody(), true);
} catch (RequestException $ex) {
return ['error' => $ex->getMessage()];
}
}
}
Summary
Manticore Search is a powerful and efficient open-source full-text search engine that excels in handling both full-text search and key-value data with ease. It is designed to scale efficiently for large datasets, providing fast search results and real-time indexing.
Manticore offers features like:
Full-text search: Allows complex and fast searches on
textual data across multiple fields.
Key-value data storage: Efficiently stores and retrieves data with a key-value mechanism.
SQL-like query interface: Supports SQL syntax for querying indexed data, which makes it easy to integrate into existing applications.
Flexible indexing: Offers advanced indexing capabilities for real-time and offline data processing.
Making a conclusion
👨👦👦 Leave a comment, I am free for discussion with your any kind technical question.