Organisiert durch Jan Lehnardt (CouchDB), Simon Willnauer (Lucene Committer) und Isabel Drost (Co-Founder & Committer of Mahout) wird sich bei der Konferenz alles rund um die Themen NoSQL, Hadoop, Lucene und weiteres aus dem Bereich Scalability drehen. Die Namen der drei dürften übrigens jedem Bekannt sein, der die ein oder andere Mailingliste rund um Lucene abonniert hat.

Neuigkeiten gibt es via @hadoopberlin sowie auf der Veranstaltungswebsite – dort werden übrigens auch noch “Helping Hands” gesucht!

• Ganglia, ein Monitoring-System für Server-Infrastrukturen nutzt ähnlich wie Chukwa Hadoop zur Analyse von Log-Files
• Avro ist ein sehr frisches Projekt von Doug Cutting und soll als Serialisierer/Store von Daten wie bspw. Pig oder Hive dienen
• Yahoo! wird die intern verwendete Hadoop-Distribution freigeben, mit der derzeit das größte Hadoop-Cluster der Welt betrieben wird (4.000 Nodes, 16 PB)
• Facebook nutzt Hadoop bzw. Hive zur Auswertung von Log-Files und zum Trainieren eines Random Forest Kategorisierers
• Cloudera bewertet die Nutzung von Hadoop auf  EC2 und die Nutzung eines datasets mit EBS inkl. Tutorial – Awesome!
• Peter Skomoroch entwickelt in 1 Woche (!) ein System (Trending Topics) zur Auswertung von Trends mithilfe von Cloudera Hadoop, Hive auf EC2

Big Data: On Cloud Computing and Hadoop from Roberto Monge on Vimeo.

The registration to ApacheCon Europe 2009 (23-27 March) in Amsterdam was opened last week. I already spent some time watching the available sessions regarding Hadoop, Lucene etc., have a look…

• Hadoop Tools and Tricks for Data Processing Pipelines
• Lucene Boot Camp
• Solr Boot Camp
• Introduction To Hadoop
• Introducing Mahout: Apache Machine Learning
• Hadoop Map-Reduce: Tuning and Debugging
• Lucene Case Studies
• Pig – Making Hadoop Easy
• Advanced Indexing Techniques with Apache Lucene
• Running Hadoop in the Cloud
• Configuring Hadoop for Grid Services
• Dynamic Hadoop Clusters
• Best of breed – httpd, forrest, solr and droids

As you can see, Apache Hadoop is playing a big role this year, and cloud computing starts to take off at Apache too. Unfortunately I didn’t find any Nutch related sessions – whats happening there?

Breakfast Links 14.01.2009

* Information Builders (Schweiz) AG übernimmt spezialisiertes Search–Team von fast, A Microsoft Subsidiary

Information Builders (ch) just acquired a 2 person team from FAST to build its new search excellence team. Their current product iWay Enterprise Index is an Enterprise Search solution and being built upon GSA and Lucene. Does anyone has any details about this combination of GSA and Lucene?

* HDFS Reliability

Tom White from Cloudera (company from US with a business model around Apache Hadoop) released a paper about the reliability of HDFS with some useful recommendations on the productive use of it.

Breakfast Links 28.11.2008

* You are a knowledge worker. Become a knowledge hero.

I just discovered the belgium company Whatever which launched a software in the field of knowledge management called Knowledge Plaza. Knowledge Plaza is focused on Enterprise Social Search methods which are roughly described here by a Whatever employee. Under the hood, Knowledge Plaza uses Apache Lucene core for Search (?) and Aperture for text extraction.

* Hadoop Map-Reduce – Tuning and Debugging

Just stumbled upon this overview for tuning and debugging Hadoop.

Unfortunately I was not be able to attend at the ApacheCon US 2008 in New Orleans this year – way too far away from good ol’ germany! But I reviewed the given sessions on Lucene (and Solr, Mahout, Tika) afterwards the conference to get some inspiration. Some comments on it:

* Advanced Indexing Techniques with Apache Lucene (Michael Busch)

Detailled presentation about Indexing capabilities of Apache Lucene and a very interesting part on how to use Token Payloads and POS-Tagging with the new TokenStream API.

* Apache Solr: Out of the Box (Chris Hostetter)

Introduction to Solr from installation and administration (Admin Console, Luke), querying (Facets, Highlighting) and configuration (Analyzers, Multiple Indexes, Replication).

* Introducing Mahout: Apache Machine Learning (Grant Ingersoll)

Already posted this session in my recent Breakfast Links. A nice presentation about what Machine Learning stands for and the approach of Mahout.

* Apache Solr: Beyond the Box (Chris Hostetter)

Presentation about Solr’s history and real world examples such like Geo search.

* Content analysis for ECM with Apache Tika (Paolo Mottadelli)

Impressive and extensive presentation about Apache Tika and its Alfresco integration for content extraction.

Breakfast Links 03.11.2008

* Exalead-Labs: voxalead

The parisian search engine company Exalead released their video search engine voxalead. It combines Exalead’s indexing (Entity Recognition) and search technology with third partie’s Voice recognition (LIMSI). Currently they indexed mostly french and us channels and a small amount of german channels. Beside the search through the voice-to-text extracted contents you can also view the original extracted text which shows off the quality of voxalead!

* Metager at Alt Search Engines

Metager now has its place at AltSearchEngines.com in english and german!

* Mahout: Twenty Newsgroups Classification

This weekend I stumbled upon a nice Mahout Classification example. It is all about creating a classification model based on twenty clusters of newsgroup messages. All you need is Mahout and Hadoop to get the example running in approx. 20 minutes.

Breakfast Links 28.10.2008

* AWS is Enterprise Ready

CloudAve summarizes the current state of Amazon’s Cloud Computing services in terms of prices, support and technologies. Amazon’s EC2 and S3 is a really exciting alternative (to hardware machines) in the large-scale processing field because of the support of Hadoop MapReduce! For german readers, there is also an up to date article at Computerwoche. I’m currently writing an article on Mozenda and GoGrid – after those I’m going to check out Amazon EC2!

* Why Defining Distance Is Important

Distance (dissimilarity) of objects explained by a real case (crime) scenario. I really appreciate those practical examples!

Breakfast Links 23.10.2008

* katta and hadoop survey slides

Katta is a project by 101tec.com a consulting and software development company specialized on large-scale data processing and information management software. Katta adds grid support to Apache Lucene with a combination of Hadoop and Zookeeper (which is going to be moved into Hadoop).

* Mozenda: SaaS Data Extraction

Mozenda is a software which is able to extract data from different sources like Websites, Databases, RSS-Feeds and more. Really interesting sounds the extraction of content from forum and blogs.

Wow, there is really too much stuff to actually test… Does anybody have any experiences in those projects/products?