News

Twitter gets new back-end search architecture

Jenny Williams

Twitter engineers have developed a new back-end search for Twitter.com with open source index search engine Lucene, to improve the micro-blogging site's search capability.

The new technology replaces Summize's relational database search engine, which was acquired by Twitter, with an inverted index search engine library written in Java. Software engineers re-wrote parts of the core in-memory data structures to meet real-time search demands.

"Scaling the old MySQL-based system had become increasingly challenging," said Twitter developer, Michael Busch, in a blog post.

"Twitter is real-time, so our search engine must be too. In addition to these scalability requirements, we also need to support extremely low indexing latencies (the time it takes between when a Tweet is tweeted and when it becomes searchable) of less than 10 seconds," said Busch.

"Our new indexer could also index roughly 50 times more Tweets per-second than we currently get," he added.

Twitter recently revamped its website to be faster, easier to use and more capable of handling photos and videos in the biggest overhaul in its four-year history.


Email Alerts

Register now to receive ComputerWeekly.com IT-related news, guides and more, delivered to your inbox.
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
 

COMMENTS powered by Disqus  //  Commenting policy