Thoughts about technology, business and all that's life.

This blog has moved to http://shal.in.

Saturday, September 26, 2009

Apache Lucene 2.9 Released


Apache Lucene 2.9 has been released. Apache Lucene is a high performance, full-featured text search engine library written entirely in Java.

From the official announce email:

Lucene 2.9 comes with a bevy of new features, including:
  • Per segment searching and caching (can lead to much faster reopen among other things)
  • Near real-time search capabilities added to IndexWriter
  • New Query types
  • Smarter, more scalable multi-term queries (wildcard, range, etc)
  • A freshly optimized Collector/Scorer API
  • Improved Unicode support and the addition of Collation contrib
  • A new Attribute based TokenStream API
  • A new QueryParser framework in contrib with a core QueryParser replacement impl included.
  • Scoring is now optional when sorting by Field, or using a custom Collector, gaining sizable performance when scores are not required.
  • New analyzers (PersianAnalyzer, ArabicAnalyzer, SmartChineseAnalyzer)
  • New fast-vector-highlighter for large documents
  • Lucene now includes high-performance handling of numeric fields. Such fields are indexed with a trie structure, enabling simple to use and much faster numeric range searching without having to externally pre-process numeric values into textual values.
  • And many, many more features, bug fixes, optimizations, and various improvements.
Look at the release announcement for more details.

Congratulations to the Lucene team! Great work as always.

This is also the last minor release which supports Java 1.4 platform. The next release will be 3.0 with which deprecated APIs will be removed and Lucene will officially move to Java 5.0 as the minimum requirement.

Solr 1.4 is not far behind and we hope to release it within two weeks.

Monday, July 06, 2009

Thoughts on Tomcat

I saw an advertisement today for taking a survey on Tomcat to help define it's future directions. I don't usually click on ads but this one seemed interesting so I did. It was a short one (thanks guys!) so I didn't mind completing it. What I did not like so much was the focus on questions on how Tomcat can compete with "enterprise" application servers. What is "enterprise" anyway? If it's about performance then Tomcat is enterprise ready. It doesn't really matter if other commercial servers can do a few more requests per second on artificial tests. Tomcat is free (as in freedom)!. That's a huge advantage.

Tomcat is the most widely used application deployment platform at AOL. As with most other web companies, we don't really need all the cruft, cost and complexity which "enterprise" application servers bring in. It is a tried and tested platform with good performance characteristics, easy administration and monitoring. It scales well enough. We run thousands of Tomcats serving hundreds of millions of pages. Since it is free, we don't need to scale by buying expensive servers, we can just scale out by adding more low cost servers (which, by the way, also adds redundancy)

I don't think Tomcat's goal should be to add features (read complexity) to compete with the so called enterprise application servers. It should continue to focus on being a performant Servlet/JSP container with easy development, administration and monitoring support. What I'd like to see added to Tomcat is:
  • Easy ways to use Tomcat in an embedded fashion (like Jetty)
  • Improve Tomcat manager
  • Easy configuration for my webapp (Properties vs. JNDI/context)
I'm not a guy who adds features just for the heck of them. So I'll give use-cases for each of the above requests. These use-cases come from my own recent work.

Easy ways to use Tomcat in an embedded fashion

I don't see myself shipping a product with an embedded tomcat but I've frequently needed to have an embedded container for unit/integration testing REST APIs. Sometimes, I've used Jetty, other times I've mocked stuff. All of my production deployments use Tomcat so it is only natural that I use Tomcat for integration testing. Solr uses Jetty for testing and for providing a standalone example which works great. I like the easy embeddable nature of Jetty. However, I also believe that part of the reason behind the popularity of Jetty was that Tomcat was not embeddable enough. It had a lot of strongly coupled extra features (and a lot of related code) which were not needed. Valves, realms, JNDI contexts, authentication and clusters are things which are generally not needed in the embeddable scenarios. Note that embedding Tomcat is possible but it is not documented that well and there is no easy way to find out all the dependency jars I'd need to do that. The last time I did this successfully with Maven, I had to track down the dependencies myself and add each one to my application to make it work. So easy for me means: publish latest jars to Maven, use the dependency structure that Maven provides, make it easy for me to remove the extra features I don't need, focus on keeping the artifacts smaller and have good documentation on how to use the API.

Improve Tomcat Manager

A few months ago, I worked on a deployment application to push code updates to Tomcat servers across data centers. The use-case is simple. I want to update my application's code without causing a downtime. So I drain traffic away from individual servers, update the code, verify that it is in fact updated, and redirect traffic back to the server. I worked with the features provided by the Tomcat manager application. Not too many people actually use the Manager in the name of security but that's a separate topic. I wanted to add some custom commands to the manager and I couldn't because it was not designed to be extensible. In the end, I had to copy code from Tomcat's sources and modify to make it work. This is an area which could use some improvement. Coupled with good documentation on how to securely use the manager application, it has the potential to be used more widely. I want to use the Tomcat manager application from certain whitelisted IPs and only with SSL. Sounds simple but it was damn hard to get it working the way we wanted.

Easy configuration for my webapp

Configuration is a difficult issue. There are always so many right ways depending on who you ask. I just need to provide some key/value pairs to my application which change rarely but when they do, I'd really like them to be reloaded without bringing my application down. I'd really like to push those into the war but then I'd have different wars for different environments (dev/qa/prod) and that'd make some people very nervous (why?). I could use JNDI but that is much more complicated to manage than it needs to be for my simple use-case. Sysadmins don't like XML, and that is a well-known fact. It's easier for everybody to modify properties files vs an XML file for simple key/value pairs. I want to hot reload them, just like Tomcat hot-loads wars dropped into the webapps directory but I guess you can't do that. So I write my own small sweet library to read properties files from a certain location, checking every few minutes for changes to the file. If Tomcat itself had something similar, I'd just use that. I think it might be a very common use-case.

On a related note, I'm very excited about tomcat-lite, comet/bayuex and the new servlet API (asynchronous servlets) coming into Tomcat. I also wish for an easy way to write non-HTTP applications on top of Tomcat's NIO stack (again de-coupling may help) but that maybe asking too much. I know it's all do-o-cracy and I'm not doing my part. Someday I hope to contribute code rather than just ideas and complaints. For now, this is all I have.

Thursday, May 28, 2009

Solr in PHP/Drupal, Ruby/Sunspot and Python/Haystack

Adoption of Apache Solr is accelerating. Being accessible though HTTP makes it possible for Solr (a Java webapp) to be used with any language. All you need is support for making HTTP calls and parsing one of the many available formats such as XML or JSON.

Drupal

Drupal is one of the most popular CMS available as open source. It is written in PHP and boasts of a huge user and developer base. Recently, the Drupal community has integrated Apache Solr into Drupal for vertical search. The integration is available as a Drupal module at http://drupal.org/project/apachesolr. There are some excellent tutorials available on how to get started with using this as well as a hosted solution by Acquia.

Ruby

Ruby integration has been present in Solr since a long time. There is a module called solr-ruby as well as acts_as_solr. Solr even has a ruby response writer which outputs search results serialized in ruby. Blacklight is an open source project I know that uses Solr and is built in Ruby. Today, I came to know about SunSpot - A Solr powered search engine for Ruby. More details at this article in LinuxMag.

Python

Solr has a python response writer as well as many clients. See http://wiki.apache.org/solr/SolPython for details. Reddit is one site that uses Solr with a python front-end application. There is also HayStack for Django which can use Solr among other engines such as Xapian and Whoosh.

Solr 1.4 is nearing release with a number of features and performance improvements. On the other hand, Lucene is getting ready for near real-time search as well. Things are getting interesting in the Solr world!

Monday, April 20, 2009

Burst of activity in Lucene

There is a large amount of work being done in Lucene 2.9, in which a large portion is related to adding support for near real-time search.

To put it very simply, search engines transfer a lot of work from query-time to index-time. The reason this is done, is to speed up queries at the cost of adding documents slower. Until now, Lucene based systems have had problems with dealing with scenarios in which the searchers need to see the changes instantly (think Twitter Search). There exist a variety of tricks and techniques to acheive this even now. However, near real-time search support in Lucene itself is a boon to all those people who have been building and managing such systems because the grunt work will be done by Lucene itself.

This is still under development and will probably take a few more months to mature. Solr will benefit from it as well but before that can happen, a lot of work will be needed under the hood particularly in the way Solr handles its caching.

Michael McCandless has summarized the current state of Lucene trunk in this email on java-dev mailing list. In fact, there is so much activity that, at times, it becomes very difficult to follow all the excellent discussions that go on. There are some very talented people on that forum and it is a lot of learning for a guy like me, who started with Solr and is still trying to find his way in the Lucene code base.

Lucene 2.9 will bring huge improvements and I'm looking forward to working with other Solr developers to integrate them with Solr.

Wednesday, April 08, 2009

Google App Engine and Maven


Google has announced support for building Java applications on the App Engine platform. This is great news for new App Engine developers and especially for those Java developers who had to learn Python to use App Engine.

I created a project for App Engine using Maven for builds. These were the steps I needed to follow:

1. Publish the App Engine libraries to the local Maven repository. Goto the app-engine-java-sdk directory (where app-engine sdk is installed) and execute the following commands:
mvn install:install-file -Dfile=lib/appengine-tools-api.jar -DgroupId=com.google -DartifactId=appengine-tools -Dversion=1.2.0 -Dpackaging=jar -DgeneratePom=true

mvn install:install-file -Dfile=lib/shared/appengine-local-runtime-shared.jar -DgroupId=com.google -DartifactId=appengine-local-runtime-shared -Dversion=1.2.0 -Dpackaging=jar -DgeneratePom=true

mvn install:install-file -Dfile=lib/user/appengine-api-1.0-sdk-1.2.0.jar -DgroupId=com.google -DartifactId=appengine-sdk-1.2.0-api -Dversion=1.2.0 -Dpackaging=jar -DgeneratePom=true

mvn install:install-file -Dfile=lib/user/orm/datanucleus-appengine-1.0.0.final.jar -DgroupId=org.datanucleus -DartifactId=datanucleus-appengine -Dversion=1.0.0.final -Dpackaging=jar -DgeneratePom=true

2. Create a maven pom file. This is the one that I used:

<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.shalin</groupId>
<artifactId>test</artifactId>
<packaging>war</packaging>
<version>1.0</version>
<name>Test</name>
<url>http://shalinsays.blogspot.com</url>

<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.google</groupId>
<artifactId>appengine-tools</artifactId>
<version>1.2.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.google</groupId>
<artifactId>appengine-local-runtime-shared</artifactId>
<version>1.2.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.google</groupId>
<artifactId>appengine-sdk-1.2.0-api</artifactId>
<version>1.2.0</version>
<scope>compile</scope>
</dependency>

<dependency>
<artifactId>standard</artifactId>
<groupId>taglibs</groupId>
<version>1.1.2</version>
<type>jar</type>
<scope>runtime</scope>
</dependency>
<dependency>
<artifactId>jstl</artifactId>
<groupId>javax.servlet</groupId>
<version>1.1.2</version>
<type>jar</type>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-el_1.0_spec</artifactId>
<version>1.0.1</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-jsp_2.1_spec</artifactId>
<version>1.0.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-servlet_2.5_spec</artifactId>
<version>1.2</version>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-jpa_3.0_spec</artifactId>
<version>1.1.1</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-jta_1.1_spec</artifactId>
<version>1.1.1</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-appengine</artifactId>
<version>1.0.0.final</version>
<scope>compile</scope>
</dependency>

</dependencies>
<build>
<finalName>test</finalName>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.5</source>
<target>1.5</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
3. Create the standard maven directory structure and add the pom.xml in the same directory as the src directory.

You're done!

I tested this with a simple servlet based application and it worked fine. I did not test the JPA/JDO integration so it might be a little rough around the edges. But it should work for the most part. Note, App Engine supports Java 6. If you want to use Java 6, you can change the "source" and "target" in the build section to 1.6 instead of 1.5

About Me

My photo
Committer on Apache Solr. Principal Software Engineer at AOL.

Twitter Updates

    follow me on Twitter

    Recently shared stories

    Recent questions on Apache Solr

    Recent development in Apache Solr