| Item |
Description |
| UTF-8 and UTF-16 support |
Index and search for Chinese, Cyrillic, Georgian, Hebrew etc. charsets. |
| Support for non-ASCII domains |
'Internationalized Domain Names' (IDN) like 'http://президент.рф/' and 'http://müller.de/' are accepted and processed. |
| Media support |
Index and search for images, audio and video. EXIF and ID3 information are also indexed and herewith become searchable, thumbnails for images, open the media with player software. |
| Multiple database support |
Individual configuration and activation of databases for Admin, Search User and Suggest URL User. Support of multiple table sets in each db, MySQL query cache, individual index for each db, individual or bulk search in predefined databases.
Individual Admin settings for each db and each set of tables. |
| Result cache |
Extremely reduced response time for queries already cached. Controller to keep the 'Most Popular Queries' always in cache. Separate caches for text and media results. Admin configurable. |
| Follow Sitemap files |
If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . >
is detected, also multiple Sitemap files are processed. |
| Periodical Re-indexing |
Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. |
| Multithreaded indexing |
In order to reduce the time for indexing, 1-10 parallel running threads are to be activated in Admin settings. |
| Erase & Re-index |
Individual (site specific) or bulk update of database. |
| RDF, RSD, RSS and Atom feed support |
Index and search of feed content, inclusive RDF 'Dublin Core' tags. Obey / ignore 'preferred' tags in RSD feeds.
Follow / ignore CDATA directives. |
| Various search modes |
Search with wildcards, Tolerant search, Search strict, Search only in one domain, Search all links of a site, Search for media (link-specific). |
9 different modes of
sorting the text results |
Admin selectable: -By relevance (weight %) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding promoted catchwords on top. |
5 different modes of
sorting the media results |
Admin selectable: -By title(alphabetic) -By file suffix
-By image size -By 'Last queried' -By 'Most popular'. |
Same results for queries typed with pure vowels
or with accents |
Will deliver the same results for queries like: cafe and café.
To be activated in Admin backend. |
| Same results for queries with and without quotes |
Admin selectable; will deliver the same results for queries like:
d'information <-> information and dei'largi <-> largi
Admin selectabe: equalization for different quotes like: ' ` ´ |
| Extensive user statistics |
Search log, Most popular text links, Most popular media links, User IP, Country code, Host name, Last queried, Top keywords, etc. |
| Segmentation of Chinese and Korean words |
Will divide phrases like 帽子和服装 into the base words 帽子 and 和 and 服装 , so that all will become searchable. Dictionaries with 106.800 radicals. |
| Segmentation of Japanese words |
Segmentation of 5.724 kanji (new, old and half width), hiragana, katakana and jinmeiyo Japanese character writing systems. |
| Transliterate Latin characters into their Greek equivalents |
Transforms query input alla to find ἀλλὰ and baptismatos to find βαπτίσματος
Also accepts queries containing Greek vowels without accents. |
| Index of password protected sites |
Index also .htaccess protected sites (basic authorization).
Up to 3 different zones could be registered and will be indexed. |
Index framesets
and iframes |
If enabled, both options will index html and image frames.
Not available for dynamically reloaded frames (e.g. JavaScript). |
| Follow HTTP redirections |
Follow URL redirections caused by HTTP 301, 302, 303 and 307 status codes. Also obeying JavaScript sent as HTML content like:
<SCRIPT language="javascript">
window.location="mp.php?mcv=59";</SCRIPT> |
| Accept gzip formatted transmission |
In order to reduce the transfer time, gzip formatted content is requested by the crawler. |
| Follow header redirections, refresh tags and cannonical links |
Automatical forwarding for the indexer. |
| Follow links found in JavaScript and index also the content |
Will index JavaScript files, detecting and following links like:
document.write(' <a href="new12.pdf">All news 2012</a> ');
and index the content of:
document.write(' this content ');
|
Index of RAR and ZIP
compressed files and archives |
Supports compressed (X)HTML, XML and also PDFs, all kind of feeds, frames and iframes in archives.
Links found in the compressed files are followed. |
Converter included for
PDF, ODT, ODS, CSV and XLS files |
Converting also non-Latin text like:
Arabic, Cyrillic, Chinese, Greece and Hebrew.
Links found in the converted files will be followed.
Separated PDF converter for 32 and 64 bit Operating Systems. |
| Debug mode |
Offering detailed information during index/re-index:
New links, keywords, frames and media found per link.
To be activated separately for Admin backend and User interface. |
Common word lists
holding stop words.
Included for 25 languages |
Admin selectable for:
Arabic, Bengali, Bulgarian, Catalan, Chinese, Cyrillic, Czech, Danish, Dutch, English, Farsi, Finnish, French, Greek, German, Hindi, Hungarian, Italian, Norwegian, Polish, Portuguese, Romanian, Spanish, Swedish and Turkish. |
Stemming algorithms
implemented for 15 languages |
Admin selectable for: Bulgarian, Chinese, Cyrillic, Czech, Dutch, English, Finnish, French, German, Greek, Hungarian, Italian, Portuguese, Spanish and Swedish. |
| Use of whitelist and blacklist |
Page must contain any / all of the words in whitelist. Admin selectable is also a blacklist, holding words to prevent indexing of pages containing these forbidden words. |
Ignore parts of a site.
<div> id/class value driven |
A common list of div id values is used to ignore parts of a page.
Content between <div id='this_value'> and </div>
as well as <div class='this_value'> and </div> will be ignored.
However links inside the tags are followed. Multiple and nested divs will be attended. |
Index only parts of a site.
<div> id/class value driven |
A common list of div id values is used to select parts of a page.
Only the content between <div id='this_value'> and </div>
as well as <div class='this_value'> and </div>; will be indexed, however links outside are followed. Multiple and nested divs will be attended. |
Indexing only parts of a page defined by
<element> . . . </element> |
Foreseen to cooperate with the new HTML5 elements like:
section, nav, aside, hgroup, article, header, footer
Vice versa function also included to ignore the parts of a page
between <element> . . . </element> |
| User may suggest URLs |
User may suggest new sites to become part of database.
With Admin approval, reject and banned domains manager.
Optional: suggested site needs meta tag with authentication code. |
| Intrusion Detection System |
Admin selectable, the IDS will block further user input, create a log file, present a warning message, or even block any traffic of IP’s known to be evil. |
| Bounded database |
Option to delete all keyword relationships, exceeding a definable amount of query results. Will significantly speed up the search procedure for huge databases. |
| XML output |
Besides the standard HTML output, optionally the results are presented as XML files. |
