Square Kilometre Array prepares for the ultimate big data challenge

Square Kilometre Array prepares for the ultimate big data challenge

The Square Kilometre Array (SKA), the world’s most powerful telescope, will be ready from day one to gather an unprecedented volume of data from the sky, even if the supporting technical infrastructure is yet to be built.

“We’ll be ready – the technology is getting there,” Bernie Fanaroff, strategic advisor for the most expensive and sensitive radio astronomy project in the world, told Science|Business.

Construction of the SKA is due to begin in 2018 and finish sometime in the middle of the next decade. Data acquisition will begin in 2020, requiring a level of processing power and data management know-how that outstretches current capabilities.

Astronomers estimate that the project will generate 35,000-DVDs-worth of data every second. This is equivalent to “the whole world wide web every day,” said Fanaroff.

The project is investing in machine learning and artificial intelligence software tools to enable the data analysis. In advance of construction of the vast telescope - which will consist of some 250,000 radio antennas split between sites in australia and South Africa - SKA already employs more than 400 engineers and technicians in infrastructure, fibre optics and data collection.

The project is also working with IBM, which recently opened a new R&D centre in Johannesburg, on a new supercomputer. SKA will have two supercomputers to process its data, one based in Cape Town and one in Perth, Australia.

Recently, elements of the software under development were tested on the world’s second fastest supercomputer, the Tianhe-2, located in the National Supercomputer Centre in Guangzhou, China. It is estimated a supercomputer with three times the power of Tianhe-2 will need to be built in the next decade to cope with all the SKA data.

In addition to the analysis, the project requires large off-site data warehouses. These will house storage devices custom-built in South Africa. “There were too many bells and whistles with the stuff commercial providers were offering us. It was far too expensive, so we’ve designed our own servers which are cheaper,” said Fanaroff.  

Fanaroff was formerly director of SKA, retiring at the end of 2015, but remaining as a strategic advisor to the project. He was in Brussels this week to explore how African institutions could gain access to the European Commission’s new Europe-wide science cloud, tentatively scheduled to go live in 2020.

 

Share it:
Share it:

[Social9_Share class=”s9-widget-wrapper”]

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

You Might Be Interested In

A New Take on Master Data Management

31 Jul, 2016

Master Data Management (MDM) is evolving. Forrester Research in its Forrester Wave: Master Data Management Q1 2016, released this spring, …

Read more

This is why dozens of companies have bought Nvidia’s $129,000 deep-learning supercomputer in a box

18 Dec, 2016

To companies grappling with complex data projects powered by artificial intelligence, a system that Nvidia calls an “AI supercomputer in …

Read more

Artificial intelligence: coming soon to a hospital near you

16 Apr, 2017

Human intelligence has long powered hospitals and health care. We rely on doctors, nurses, and a variety of other clinicians …

Read more

Recent Jobs

IT Engineer

Washington D.C., DC, USA

1 May, 2024

Read More

Data Engineer

Washington D.C., DC, USA

1 May, 2024

Read More

Applications Developer

Washington D.C., DC, USA

1 May, 2024

Read More

D365 Business Analyst

South Bend, IN, USA

22 Apr, 2024

Read More

Do You Want to Share Your Story?

Bring your insights on Data, Visualization, Innovation or Business Agility to our community. Let them learn from your experience.

Get the 3 STEPS

To Drive Analytics Adoption
And manage change

3-steps-to-drive-analytics-adoption

Get Access to Event Discounts

Switch your 7wData account from Subscriber to Event Discount Member by clicking the button below and get access to event discounts. Learn & Grow together with us in a more profitable way!

Get Access to Event Discounts

Create a 7wData account and get access to event discounts. Learn & Grow together with us in a more profitable way!

Don't miss Out!

Stay in touch and receive in depth articles, guides, news & commentary of all things data.