How to get experience working with large data sets

http://www.philwhln.com/how-to-get-experience-working-with-large-datasets

Quora

www.quora.com/Data/Where-can-I-find-large-datasets-open-to-the-public -- very good collection of links

DataMobs

http://datamob.org/datasets

KDNuggets

http://www.kdnuggets.com/datasets/

Research Pipeline

http://www.researchpipeline.com/mediawiki/index.php?title=Main_Page

Google Public Data Directory

http://www.google.com/publicdata/directory

Delicious 1

https://delicious.com/pskomoroch/dataset

Delcious 2

https://delicious.com/judell/publicdata?networkaddconfirm=judell

Public Data sets on Amazon AWS

Amazon provides following data sets : ENSEMBL Annotated Gnome data, US Census data, UniGene, Freebase dump

Data transfer is 'free' within Amazon eco system (within the same zone)

AWS data sets

InfoChimps

InfoChimps has data marketplace with a wide variety of data sets.

InfoChimps market place

Comprehensive Knowledge Archive Network

open source data portal platform

data sets available on datahub.io from ckan.org

Stanford network data collection

http://snap.stanford.edu/data/index.html

Open Flights

Crowd sourced flight data http://openflights.org/

Flight arrival data

http://stat-computing.org/dataexpo/2009/the-data.html

Wikipedia data

wikipedia data

OpenStreetMap.org

OpenStreetMap is a free worldwide map, created by people users. The geo and map data is available for download.
openstreet.org

Natural Earth Data

http://www.naturalearthdata.com/downloads/

Geocomm

http://data.geocomm.com/drg/index.html

Geonames data

http://www.geonames.org/

US GIS Data

Available from http://libremap.org/

Wikipedia data

wikipedia data

Google N-gram data

google ngram

Public terabyte data

Web data crawl data linky

Freebase data

variety of data available from http://www.freebase.com/

Stack OverFlow data

http://blog.stackoverflow.com/category/cc-wiki-dump/

UCI KDD data

http://kdd.ics.uci.edu/

European Parliament proceedings

proceedings from Statistical machine Translation

US government data

data.gov

UK government data

data.gov.uk

US Patent and Trademark Office Filings

www.google.com/googlebooks/uspto.html

World Bank Data

http://datacatalog.worldbank.org/

Public Health Data sets

http://phpartners.org/health_stats.html

National Institute of Health

http://projectreporter.nih.gov/reporter.cfm

Aid information

http://data.devinit.org

UN Data

http://data.un.org/Explorer.aspx