San Francisco-based online residential real estate company Trulia lives and dies by data. To compete successfully in today's housing market, tt must deliver the most up-to-date real estate information available to its customers. But until recently, doing so was a daily struggle.
Acquired by online real estate database company Zillow in 2014 for $3.5 billion, Trulia is one of the largest online residential real estate marketplaces around, with more than 55 million unique site visitors each month.
With so much data to store and process, the company adopted Hadoop in 2008 and it has since become the heart of Trulia's data infrastructure. The company has expanded usage of Hadoop to an entire data engineering department consisting of several teams using multiple clusters. This allows Trulia to deliver personalized recommendations to customers based on sophisticated data science models that analyze more than a terabyte of data daily. That data is drawn from new listings, public records and user behavior, all of which is then cross-referenced with search criteria to alert customers quickly when new properties become available.
To make it all work, throughout each night, the company must complete dozens of workflows and hundreds of complex jobs on time. With many teams writing Hadoop jobs or using Hive or Spark concurrently, Trulia has to ensure reliability in its multi-tenant, multi-workload environment. Delayed or unpredictable jobs throw a wrench in the works and can seriously affect the bottom line. Until recently, that meant Trulia had to intentionally underutilize its Hadoop clusters to ensure jobs completed on time.
"We process, on a daily basis, over a terabyte of new information: public records, listings, user activity," says Zane Williamson, senior director of DevOps at Trulia. "We process this data across multiple Hadoop clusters and use the information to send out email and push notifications to our users. That's the lead driver to get users back to the site and interacting. It's very important that it gets done in a daily fashion. Reliability and uptime for the workflows is essential."
"It's been a pretty painful process, I think," Williamson adds, noting that he joined Trulia relatively recently. "It's been a pretty big challenge to reliably run this data cycle, maintain uptime and troubleshoot issues.
