Get that Search Button Working with Apache Solr!

Solr is an open source search server that was originally developed at CNet, and then donated to Apache. You should only be thinking about Solr if you want to implement some kind of search facility for your Web application that’s backed by random sources of data. Although it has been built with enterprise applications in mind, where users would set up instances of a search server on a separate machine dedicated for this purpose, this shouldn’t stop you from using it for small projects too.

Solr is based on the full-text search library called Apache Lucene. It is mostly built as an enhancement on top of an index provided by it. Solr has been built completely in Java and so, you’ll need to have Java and a servlet container like Tomcat or GlassFish installed, on top of which Solr can be deployed as a war package. However, that does not mean that you need to know Java, unless you want to make modifications to Solr!

Solr provides XML/HTTP interfaces and APIs just like Web services, and so can work with almost any programming language. The extensive external configurations available are more than enough to make customisations. Apart from this, it even provides a comprehensive administration interface with a browser.

Solr also gives you a good set of default settings to start with, and provides good documentation along with it. The best applications being used out there are highly customised, but they require a deep understanding of the type of data to be searched, and the queries that users are going to make.

Information can be collected directly from filesystems, databases, websites, directories, mail servers or anything else. On the whole, Solr is a good compromise for enterprise use as well as personal applications.

Why Solr?

Whenever we think about search, what comes to our minds is usually Google; after all, it has been the world leader in terms of search technology. Then why use Solr?

For starters, Solr allows you to have precise control over what gets indexed, how it gets indexed and how it’s retrieved from the index. In short, a powerful search solution that is totally under your control, and can change the way users perceive and interact with your application. Setting up a separate instance of a search server will help you streamline your website and help your users wade through data with an ease that they have never experienced before.

But then, why go through the trouble of setting up a separate search server, when even databases can provide such text-indexing capabilities? Why can’t we just SELECT * FROM Content to retrieve data? More importantly, if Solr has been implemented on top of Lucene, why can’t we build our own search application with its help?

Well, building a search application on top of Lucene would require a lot of work — and you’d really have to know what you’re doing (in which case, you wouldn’t need to read this article at all). Talking about comparisons with databases, it depends on what kind of search application you want to build. If you want something that helps with exact-match queries, like the names of all the people who were born in 1988, or a list of all authors who wrote books on a particular subject, then you’re better off with databases, without any extra effort. However, if you want to answer queries that are less predictable and need proper analysis, like a list of documents in which a particular keyword occurs, or anything similar that is more flexible, Solr is there for you.

Before creating a database in a DBMS, you need to properly design a schema — and even minor changes in the structure of the data need to be reflected back into the database. This is not so with Solr — actually, it acts more like a NoSQL document store, where documents are in turn stored in the form of a Lucene index. Does that mean we can use it as a NoSQL database?

Probably, but Solr hasn’t been built for that purpose; it’s more efficient as a search server, and we should let it remain that way.

Features and components

Carefully implemented instances of Solr can provide features that can match even those of Google — and what’s more, it allows for further customisation, if needed; after all, it’s open source!

Let’s take a brief look at some of the important and special features that Solr has to offer, without going into too much technical detail.

• Faceted search: This is one feature that stands out from the others. Whenever we encounter search in most applications, it comes with options to narrow down our search results with respect to certain parameters. Faceting provides such functionality, by providing facets as the navigational elements in the search. This technique is quite popular in places like online e-commerce websites, where we might need to filter search results from a particular category or a particular company.

  

• Spatial search: Although spatial search or geo-aware search is not a “feature” as such, it is quite relevant these days — and geo-location information is often included with the data, so it is important to mention it here. Spatial search is a plugin that is expected to be included with native support in Solr version 4.0, and will provide results filtered according to the location information. For example, a search for a coffee shop should produce results suitable to your current location.
• Clustering: Clustering is also used for the aggregation of data. Solr provides for search result clustering with the Carrot2 clustering engine. Clustering is different from faceting, in the sense that it is done dynamically. We don’t need to mark documents or specify fields to be categorised for clustering to take place; it is done automatically by recognising the words that occur most commonly in the text, and by recognising the structure of the text. Clustering is done when the search query is executed, while faceting is done during indexing. One of the clustering algorithms that I am aware of is the k-means clustering algorithm.

Other features (which search engines must obviously have) include hit highlighting, auto-suggest, spell-checker and the “more like this” functionality, which do not need introduction.

Installation and setup

First of all, let’s get down to installing and setting up a single core instance of Solr on top of Tomcat. We will try to install the example schema that comes along with the archive.

Before proceeding, it is worth mentioning that both the Lucene and Solr projects were merged in March 2010, due to which the version numbering sometimes becomes quite ambiguous. To avoid such confusion, let’s look at the current and next version numbers of Lucene/Solr:

So, having dealt with that, you can begin by downloading and unzipping the apache-solr-1.4.1.tgz file from the website. As mentioned before, you will need to have at least Java 5 and a servlet container installed.

In the apache-solr-1.4.1/dist directory, you will find the apache-solr-1.4.1.war file, which can be uploaded to any servlet container (in our case, I assume it is Tomcat). Rename it to solr.war and deploy the file.

Next, you need to set up the home folder which contains all the configuration files, etc. To do this, you can either go to apache-solr-1.4.1/example, copy the solr directory, and paste it into the bin folder of Tomcat — or you can set the Java system property of the solr.solr.home parameter to point to the location of your home folder.

Now, visit http://localhost:8080/solr/admin. You should see the Solr Admin console (see Figure 2), and you’re done.

If you’re still not able to get this working, you can try an example setup using the lightweight Jetty server, which can be run using the following command in the example folder:

java -jar start.jar

This will set up Jetty to listen on port 8983 instead of 8080, so you will have to replace the port number in the corresponding examples. You may explore the Admin console now, which consists of several tabs and a query box. At this point, queries will not return results, since we haven’t yet indexed anything; we will do so later.

Architecture

Solr consumes data in the form of documents. A document is like a basic unit of information that is fed into Solr. It consists of fields, which are, in turn, a more specific piece of information.

The process shown so far, is a simplified version of the installation procedure, and probably not how you would like it in a production system. For such purposes, you should refer the actual documentation on the Solr wiki.

Let’s now look at some of the important files, and the directory structure:

The two most important files in the conf directory are as follows; each of these is extensively commented, with every parameter explained in detail:

• solrconfig.xml — This file handles the entire primary configuration. It is used to define request handlers, response writers (explained below), auto-warming parameters and plugins.
• schema.xml — This file, as is obvious, dictates the “schema” or the structure of your documents. It also specifies how each field in the document should be indexed, and what fields each document can contain. The file is divided into sections to define types, fields and other definitions pertaining to documents. It also specifies Analyzers in order to determine the manner in which each field is indexed and retrieved. Schema design is one of the most important steps while designing an application on top of Solr.

RequestHandler

A RequestHandler dictates what is to be done when a particular request is made. It basically processes all the interactions with the search server, including querying, indexing, etc. Examples of some of the many RequestHandlers available (besides the StandardRequestHandler) are DisMaxRequestHandler, DataImportHandler, MoreLikeThisHandler, SpellCheckerRequestHandler, etc.

Each of these are built to perform specific functions; Solr even allows you to write custom handlers for your own purpose.

ResponseWriter

ResponseWriter generates and formats the response to a query, or any request made to the server. Multiple ResponseWriters are available to produce the data to be consumed in the form of JSON, PHP, Python, Ruby, CSV, XML and even XSLT. A custom ResponseWriter can also be written in a manner similar to a RequestHandler, by implementing an interface specified for each, in Java.

Search components

Search components are something that provide a specific functionality that can be reused with any type of handler to provide a common logic. These are the most basic extension methodologies in Solr. Each component is configured in the solrconfig.xml file as an addition to the RequestHandlers.

Indexing techniques

Indexing is feeding data into Solr’s index. Proceeding with the example, let’s first send the data to our installed application for indexing, with the help of the included simple post application. Go to the apache-solr-1.4./example/exampledocs folder, from the location where you initially unzipped the Solr archive. In this directory, give the following command:

java -Durl=http://localhost:8080/solr/update -jar post.jar *.xml

After the process is completed, you will be able to search for your data from the admin console. The data that gets posted to the Solr index is all the XML files that are contained n the exampledocs folder. You may look at the structure of each of the documents that get posted.

There are multiple ways for sending data to be indexed:

• XML-post: Solr allows for a predefined set of XML schemas, which specify the instruction or the action to be performed. This may range from the simple addition of new documents, to deletion, updation and committing of data. These XML commands can simply be POSTed to the server (at http://localhost:8080/solr/updatein this case), and it will perform the specified operation. This is precisely what the post.jar application does. For example, to delete a document, you can run the following command in a terminal:

  curl http://localhost:8080/solr/update -F stream.body='<delete><id>VA902B</id></delete>'

• CSV: Solr can import the data to be accepted in the form of a CSV format, where values are usually separated by a delimiter such as a comma or a tab. It can also be set up to accept streams of data in the form of CSV, if your data source generates it. CSV data can be sent to Solr by sending data to http://localhost:8080/solr/update/csv. A sample books.csv file is included in exampledocs, which can be uploaded using curl in a manner similar to that shown above.
• Solr Cell: What more could you possibly want from a search server, than its ability to index PDF, Word, Excel, JPEG and even MP3 files? Solr Cell, a.k.a. ExtractingRequestHandler, is what allows you to do so! It has been built as a wrapper around the Apache Tika project, which can strip out data from a wide range of supported documents — and what’s more, it can even auto-detect the MIME type of the data.
• DataImportHandler: This has been built to handle direct database imports, either with the help of JDBC, or XML imports from a URL or a file. The XML import functionality can be used to index data over HTTP/REST APIs, or even RSS feeds. In addition to full import, it supports delta-imports — that is, a partial update of the index, reflecting only the recent changes in the database since the last build, instead of a complete rebuild of the index. The DataImportHandler can be defined in the solrconfig.xml file; it can further point to a file for detailed configuration of the connection and data source information. The example-DIH folder already contains some examples, along with detailed documentation, for you to try out.

Querying

So finally we have all our data set, and we are ready to make a query or two in order to know how it works. Let’s make a basic query now: type solr into the search box at the admin console, and see what you get. The URL generated looks something like what’s shown below:

http://localhost:8080/solr/select/?q=solr&version=2.2&start=0&rows=10&indent=on

We can guess that http://localhost:8080/solr/select is the endpoint at which we will be making our queries, while the q=solr parameter represents our search query. The start=0 and rows=10 parameters are used for paginating, since Solr does not display all results at once. Besides this, there are a lot of parameters that have been left out, which include features like boosting, sub-expressions, filter queries, sorting, and of course, faceting!

Additionally, the DisMax Query parser allows for even more sophisticated queries that should be more than enough to satisfy even advanced users.

Looking at the search results, we’ll be able to see XML that has been enclosed within the <response> tags. The initial metadata, like the status, query time, etc., are enclosed within the <lst name="responseHeader"> element, and the subsequent results are enclosed within the <result name="response" numFound="1" start="0"> element, which further contains the <doc> element that represents each document returned as a result. It should be obvious that the numFound attribute represents the number of results returned.

In conclusion, it will be useful to note that add-on modules for using Solr with CMSes like Drupal and WordPress are already available. These modules include all the configuration information to set up Solr for your website, and provide a much better searching mechanism than the one included with the core.

Further reading

• Solr Wiki
• Indexing Data in Solr 1.4 Enterprise Search Server
• Search Smarter with Apache Solr