Review: Surviving the data tsunami

For the 16th meeting of the astronomy twitter journal club we were looking for ways to prevent us all drowning in data when upcoming instruments and surveys get going. The discussion was based on this paper, and you can also read our preview and the full transcript of the meeting.

Onto the review…

So, do you agree with the paper’s authors that we’re facing a ‘data tsunami’? – @astronomyjc

From what I’ve heard about LSST in particular: yes, a data tsunami might well be upcoming! – @MarcelAstroph

i think that the earthquake to trigger the tsunami hasn’t quite happened yet. things like WFIRST and LSST actually go on the sky.. actually I take that back. LOFAR, VISTA, etc are on the sky – @augustmuench

Yes data tsunami is at the doorstep. Probably each of us have couple of 10ths Gigs data on our own – @khanzadyan

But the increase from SDSS to LSST is slower than the (current) pace of Moore’s law…A quick plot w/ Python of the current data increase in optical surveys. The gray dashed line is Moore’s law.

That looks convincing, but I’m still worried about the other wavelengths i.e radio, submm – @astronomyjc

I agree radio can produce a ridiculous amount of data if everything was recorded 🙂 Some day tho … – @GeertMcTwit

Are we making adequate preparations for it then? Will we be able to handle the data? – @astronomyjc

I think we mostly need new education and/or funding policies: either train or hire computer scientists. – @MarcelAstroph

I would say we need more pre-processing and final product. I’m sure these can be separated from the main archive – @khanzadyan

Training Computer Scientists with astro is fine by me 😉 – @kashfarooq

Should we be investing more in projects like the Virtual Observatory which bring together multiple online datasets? – @astronomyjc

Most definitely yes! We need to know what data is out there to prevent multiple almost identical data-sets – @khanzadyan

Unified data archives can also be good for stimulating people to use archival data rather than expensive observatory time… – @MarcelAstroph

lots of people use 2MASS data. A good archive with documentation is key. – @johngizis

What’s the next step? Apply for funding? Introduce phd IT education? Buy more servers? ?? – @astronomyjc

would say collaborate in VO type of initiative. Would be more convincing for policy makers – @khanzadyan

I think training in astro is difficult enough. Why not team up w/ compsci? Let people do what they’re good at? Interdisciplinarity! – @MarcelAstroph

next steps: training? that requires buy in from advisors & students. what if students say, “we’ll figure it out on our own!” – @augustmuench

lots of developers are always looking for interesting open source projects to be involved with. – @kashfarooq

I’d say decent programming skills are as important a tool as maths – it’s time for Physics courses to include it! – @GeertMcTwit

The final word:

some would say & I agree that “computer science” != “scientific computing” we should teach the latter and partner with the former. – @augustmuench

This entry was posted in Reviews and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s