Twitter gets new back-end search architecture


Twitter gets new back-end search architecture

Jenny Williams

Twitter engineers have developed a new back-end search for with open source index search engine Lucene, to improve the micro-blogging site's search capability.

The new technology replaces Summize's relational database search engine, which was acquired by Twitter, with an inverted index search engine library written in Java. Software engineers re-wrote parts of the core in-memory data structures to meet real-time search demands.

"Scaling the old MySQL-based system had become increasingly challenging," said Twitter developer, Michael Busch, in a blog post.

"Twitter is real-time, so our search engine must be too. In addition to these scalability requirements, we also need to support extremely low indexing latencies (the time it takes between when a Tweet is tweeted and when it becomes searchable) of less than 10 seconds," said Busch.

"Our new indexer could also index roughly 50 times more Tweets per-second than we currently get," he added.

Twitter recently revamped its website to be faster, easier to use and more capable of handling photos and videos in the biggest overhaul in its four-year history.

Email Alerts

Register now to receive IT-related news, guides and more, delivered to your inbox.
By submitting your personal information, you agree to receive emails regarding relevant products and special offers from TechTarget and its partners. You also agree that your personal information may be transferred and processed in the United States, and that you have read and agree to the Terms of Use and the Privacy Policy.

COMMENTS powered by Disqus  //  Commenting policy