Datasets

Machine Learning Datasets

Introduction:

Without Data there is no Machine Learning, no AI, no Deep Learning. Because of heavy automation, IOT devices all around, there is no dirth of data. The first issue is, due to privacy and security related issues, data is not available for everyone. The second issue is cleaning this data. He third issues is getting complete data which can solve a given business problem. To get the complete data you need to get the data from multiple sources, identify the key to connect different records/sample of different sources. It is an expensive and time-consuming step of data science project. If you want to learn data science or want to solve any existing problem using new methods. Then you need some benchmarking framework in place which can display the model metrics (recall/precision/accuracy etc.) of each new approach (algorithm) against a given dataset. So datasets play a critical role in benchmarking algorithm performance.

Thus, machine learning rely heavily on high-quality datasets for training and evaluation. These datasets serve as the foundation for developing robust and accurate models across various AI domains, including classical machine learning, computer vision, natural language processing (NLP), audio processing, and time series analysis. Access to diverse and comprehensive datasets is crucial for researchers and practitioners to tackle real-world problems and advance the field of machine learning.

In this article, I am publishing a curated collection of datasets sourced from over 150 data sources. These datasets and data sources have been carefully selected to cover a wide range of domains, ensuring their relevance to different machine learning applications. Whether you are working on a NLP/text project, CV/image project, audio project, or time series forecasting, or classical machine learning, you’ll find valuable datasets to support your research and development efforts. Let’s dive into the world of machine learning datasets and discover the wealth of resources available to fuel your projects. If you dig each link, you will find hundreds, if not thousands of datasets under many of the links shared. I hope you will get benefitted from this work. These links I got from chrome bookmarks. At the time of writing this article, I have validated the link. If you find any link is not work / wrongly pointing / wrongly describing then please help me in improving this article. You can write to me at hari.prasad @ vedavit-ps .com.

List of Datasets and Data Sources

Sno.	URL	Description
1.	100+ Interesting Data Sets for Statistics	100+ Interesting Data Sets for Statistics
2.	100+ Mammography Image Databases	Mammography Image Databases – 100 or more images of mammograms with ground truth. Additional images available by request, and links to several other mammography databases are provided. (Formats: homebrew)
3.	15 amazon datasets on data.world	amazon data on data.world	8 datasets available
4.	20 Free Big Data Sources	20 Free Big Data Sources
5.	332 Sport Datasets on data.world	sports data on data.world	338 datasets available
6.	4000+ Groningen Natural Image Database	Groningen Natural Image Database – 4000+ 1536×1024 (16 bit) calibrated outdoor images (Formats: homebrew)
7.	450+ UCI datasets
8.	538 Datasets	FiveThirtyEight: data and code related to their articles
9.	538 Datasets Summary Githu	FiveThirtyEight: data and code related to their articles
10.	57 products datasets on data.world
11.	622 UCI Archive Dataset	UCI Archive-Machine Learning Repository: Data Sets
12.	A Collective list of Free API for Datasets	A collective list of free APIs for use in software and web development.
13.	Academic Torrents- Large Research dataset	Academic Torrents: distributed network for sharing large research datasets
14.	The Air Freight data set is a ray-traced image sequence	Air Freight - The Air Freight data set is a ray-traced image sequence along with ground truth segmentation based on textural characteristics. (455 images + GT, each 160x120 pixels). (Formats: PNG)
15.	Air Freight Dataset - Computer Vision	Air Freight – The Air Freight data set is a ray-traced image sequence along with ground truth segmentation based on textural characteristics. (455 images + GT, each 160×120 pixels). (Formats: PNG)
16.	Allen Institutes Dataset	Datasets – Allen Institute for AI
17.	Amazon Datasets	Amazon Web Services Public Data Sets
18.	Amsterdam Library of Object Images - ALOI	Amsterdam Library of Object Images – ALOI is a color image collection of one-thousand small objects, recorded for scientific purposes. In order to capture the sensory variation in object recordings, we systematically varied viewing angle, illumination angle, and illumination color for each object, and additionally captured wide-baseline stereo images. We recorded over a hundred images of each object, yielding a total of 110,250 images for the collection. (Formats: png)
19.	Annotated face, hand, cardiac & meat images	Annotated face, hand, cardiac & meat images – Most images & annotations are supplemented by various ASM/AAM analyses using the AAM-API. (Formats: bmp,asf)
20.	Apigee	Apigee: explore dozens of popular APIs
21.	apilist.fun	API List: A public list of free APIs for programmers
22.	AT&T Laboratories Cambridge face database	AT&T Laboratories Cambridge face database
23.	AVHRR Pathfinder	AVHRR Pathfinder
24.	Awesome Deep Learning Database	Densely Sampled View Spheres – Densely sampled view spheres – upper half of the view sphere of two toy objects with 2500 images each. (Formats: tiff)
25.	Awesome Public Dataset
26.	Awesome Public Datasets	Awesome Public Datasets: Well-organized and frequently updated
27.	B2SHARE
28.	Berkeley Segmentation Dataset 500	Berkeley Segmentation Dataset 500
29.	Biometric Systems Lab	Biometric Systems Lab – University of Bologna
30.	Caltech Image Database	Caltech Image Database – about 20 images – mostly top-down views of small objects and toys. (Formats: GIF)
31.	CAVIAR video sequences of mall and public space behavior	CAVIAR video sequences of mall and public space behavior - 90K video frames in 90 sequences of various human activities, with XML ground truth of detection and behavior classification (Formats: MPEG2 & JPEG)
32.	CCITT Fax standard images	CCITT Fax standard images – 8 images (Formats: gif)
33.	Census of India
34.	Global Terrorism Database (GTD)
35.	CIFAR-10 and CIFAR-100	CIFAR-10 and CIFAR-100
36.	CMU CIL’s Stereo Data (Image)	CMU CIL’s Stereo Data with Ground Truth – 3 sets of 11 images, including color tiff images with spectroradiometry (Formats: gif, tiff)
37.	CMU PIE Database	CMU PIE Database - A database of 41,368 face images of 68 people captured under 13 poses, 43 illuminations conditions, and with 4 different expressions.
38.	CMU VASC Image Database	CMU VASC Image Database – Images, sequences, stereo pairs (thousands of images) (Formats: Sun Rasterimage)
39.	College Scorecard Data
40.	Columbia-Utrecht Reflectance and Texture Database	Columbia-Utrecht Reflectance and Texture Database – Texture and reflectance measurements for over 60 samples of 3D texture, observed with over 200 different combinations of viewing and illumination directions. (Formats: bmp)
41.	Computational Colour Constancy Data	Computational Colour Constancy Data - A dataset oriented towards computational color constancy, but useful for computer vision in general. It includes synthetic data, camera sensor data, and over 700 images. (Formats: tiff)
42.	Computational Vision Lab	Computational Vision Lab
43.	Content-based image retrieval database	Content-based image retrieval database - 11 sets of color images for testing algorithms for content-based retrieval. Most sets have a description file with names of objects in each image. (Formats: jpg)
44.	Cricket Data
45.	Crowdanalytics
46.	Crowdflower Dataset	CrowdFlower: interesting datasets created or enhanced by their contributors
47.	Crunchbase	Crunchbase: Discover innovative companies and the people behind them
48.	CVD Foundation Open Images	Open Images dataset – Open Images is a dataset of ~9 million URLs to images that have been annotated with labels spanning over 6000 categories.
49.	Data world
50.	Data.lacity.org	DataLA
51.	DataCamp
52.	DataInnovation Dataset Blog	Center for Data Innovation: blog posts about interesting, recently-released data sets.
53.	Dataverse.org	Dataverse Project: searchable archive of research data
54.	DC Open Data Catalog	DC Open Data Catalog / OpenDataDC
55.	Deep Fashion	Large-scale Fashion (DeepFashion) Database – Contains over 800,000 diverse fashion images. Each image in this dataset is labeled with 50 categories, 1,000 descriptive attributes, bounding box and clothing landmarks
56.	Devanagari Handwritten Character Dataset
57.	Donor Choose	Donors Choose: data related to their projects
58.	Face and Gesture images and image sequences	Face and Gesture images and image sequences – Several image datasets of faces and gestures that are ground truth annotated for benchmarking http://www.fg-net.org/
59.	FG-NET Facial Aging Database	FG-NET Facial Aging Database – Database contains 1002 face images showing subjects at different ages. (Formats: jpg)
60.	Finding Datasets	inside-R
61.	Flickr 30k	Flickr 30k
62.	Flickr 8k	Flickr 8k
63.	Flickr Data 100 Million Yahoo dataset	Flickr Data 100 Million Yahoo dataset
64.	FVC2000 Fingerprint Databases	FVC2000 Fingerprint Databases - FVC2000 is the First International Competition for Fingerprint Verification Algorithms. Four fingerprint databases constitute the FVC2000 benchmark (3520 fingerprints in all).
65.	Gapminder Data
66.	German Fingerspelling Database	German Fingerspelling Database – The database contains 35 gestures and consists of 1400 image sequences that contain gestures of 20 different persons recorded under non-uniform daylight lighting conditions. http://www-i6.informatik.rwth-aachen.de/~dreuw/database.html
67.	Getting Stock Data
68.	Github-DataMeet	Datameet is a community of Data Science enthusiasts.
69.	Google House Numbers from street view
70.	Google Scholar
71.	HowStat	HowSTAT! The Cricket Statisticians – Home Page
72.	github.com/TheUpShot	The Upshot: data related to their articles
73.	Huggingface datasets
74.	Humanitarian Data Exchange	Humanitarian Data Exchange
75.	IEEE DataPort	Data Competitions	IEEE DataPort
76.	IEEN Image Library	IEN Image Library – 1000+ images, mostly outdoor sequences (Formats: raw, ppm)
77.	Image Analysis Laboratory	Image Analysis Laboratory – Images obtained from a variety of imaging modalities — raw CFA images, range images and a host of “medical images”. (Formats: homebrew)
78.	Image QA	Image QA
79.	ImageNet	ImageNet
80.	IMDb Top 250 Movies	Ratings and Reviews for New Movies and TV Shows – IMDb
81.	IMF-Exchange Rate	IMF-Exchange Rate Archives by Month
82.	Indian Govt
83.	Indian Liver Patient Dataset
84.	INRIA	INRIA
85.	Institute of Computer Graphics and Vision	Institute of Computer Graphics and Vision
86.	Inter University Consortium for Politics & Social	Inter-university Consortium for Political and Social Research
87.	Kaggle Datasets	Kaggle provides datasets with their challenges, but each competition has its own rules as to whether the data can be used outside of the scope of the competition.
88.	kdnuggets	kdnuggets- Datasets for Data Mining and Data Science
89.	Mammography Image Databases	Mammography Image Databases - 100 or more images of mammograms with ground truth. Additional images available by request, and links to several other mammography databases are provided. (Formats: homebrew)
90.	Mashape - Explore APIs	Mashape: explore hundreds of APIs
91.	Microsoft COCO	Microsoft COCO
92.	Million Song Dataset	Million Song Dataset
93.	MIT Vision Texure	MIT Vision Texture – Image archive (100+ images) (Formats: ppm)
94.	MNIST Handwritten digits	MNIST Handwritten digits
95.	NLM HyperDoc Visible Human Project	NLM HyperDoc Visible Human Project - Color, CAT and MRI image samples - over 30 images (Formats: jpeg)
96.	NYC Open Data socrata	NYC Open Data
97.	OASIS 1	OASIS-1 (Open Access Series of Imaging Studies)
98.	OASIS Brain - Imaging Studies	Cross-Sectional MRI Data in Young, Middle Aged, Nondemented, and Demented Older Adults
99.	Open Images is a dataset of ~9 million URLs	Open Images dataset - Open Images is a dataset of ~9 million URLs to images that have been annotated with labels spanning over 6000 categories.
100.	Photometric 3D Surface Texture Database	Photometric 3D Surface Texture Database - This is the first 3D texture database which provides both full real surface rotations and registered photometric stereo data (30 textures, 1680 images). (Formats: TIFF)
101.	Pittsburgh Science of Learning	Pittsburgh Science of Learning Center’s DataShop
102.	ProPublica Data Store	ProPublica Data Store
103.	Python API for Datasets	Python APIs: Python wrappers for many APIs
104.	Quanddl	Quandl: over 10 million financial, economic, and social datasets
105.	R Datasets	Rdatasets: collection of 700+ datasets originally distributed with R packages
106.	RapidAPI.com	25 Free Public APIs for Developers & Free Alternatives List
107.	rdatamining.com	RDataMining.com
108.	Reddit Dataset	Datasets subreddit: ask for help finding a specific data set, or post your own
109.	Reddit Dataset from 2500 subreddits	Reddit Top 2.5 Million: all-time top 1,000 posts from each of the top 2,500 subreddits
110.	Reddit Dataset Jeopardy Question	200,000+ Jeopardy questions
111.	research.yahoo.com
112.	Sebastian Raschka	Sebastian Raschka: datasets categorized by format and topic
113.	Smartcities Data Govt of India
114.	Stanford Edu Dataset	Stanford Large Network Dataset Collection: graph data
115.	Suicide Rates 1985-2013	Suicide Rates Overview 1985 to 2016	Kaggle
116.	Sunlight Foundation Govt Data	Sunlight Foundation: government-focused data
117.	India, Surat City
118.	UP Govt Economics	Directorate of Economics and Statistics UP Govt.
119.	UP Smart Cities
120.	Tamilnadu	37K Resources, 4,134 Catalog, 101 Departments
121.	The MIT-CSAIL Database of Objects and Scenes	The MIT-CSAIL Database of Objects and Scenes - Database for testing multiclass object detection and scene recognition algorithms. Over 72,000 images with 2873 annotated frames. More than 50 annotated object classes. (Formats: jpg)
122.	Tiny Images 80 Million tiny images	Tiny Images 80 Million tiny images6.
123.	Traffic Image Sequences and ‘Marbled Block’ Sequence	Traffic Image Sequences and ‘Marbled Block’ Sequence - thousands of frames of digitized traffic image sequences as well as the ‘Marbled Block’ sequence (grayscale images) (Formats: GIF)
124.	U Oulu wood and knots database	U Oulu wood and knots database - Includes classifications - 1000+ color images (Formats: ppm)
125.	UC Irvine Machine Learning Repository	UC Irvine Machine Learning Repository
126.	UCI	UC Irvine Machine Learning Repository: datasets specifically designed for machine learning
127.	UCI Archive 620+ datasets
128.	UCI-Liver Disorder Datasets	UCI Machine Learning Repository: Liver Disorders Data Set
129.	UFO - Geolocation and Time Dataset	UFO reports: geolocated and time-standardized UFO reports for close to a century
130.	UK Govt	data.gov.uk
131.	University of Oulu Physics-based Face Database	University of Oulu Physics-based Face Database - contains color images of faces under different illuminants and camera calibration conditions as well as skin spectral reflectance measurements of each person.
132.	University of Oulu Texture Database	University of Oulu Texture Database - Database of 320 surface textures, each captured under three illuminants, six spatial resolutions and nine rotation angles. A set of test suites is also provided so that texture segmentation, classification, and retrieval algorithms can be tested in a standard manner. (Formats: bmp, ras, xv)
133.	US Census Bureau	US Census Bureau
134.	US Gov 256K datasets	The Home of the U.S. Government’s Open Data
135.	US Govt	data.gov (see also: Project Open Data Dashboard)
136.	US Students Univerties
137.	USF Range Image Data with Segmentation	USF Range Image Data with Segmentation Ground Truth - 80 image sets (Formats: Sun rasterimage)
138.	Vanderbilt edu dataset websites
139.	Vanderbilt edu datasets
140.	VQA	Visual Question Answering
141.	Wikipedia Dataset	Wikipedia:Database download – Wikipedia
142.	Wiry Object Recognition Database	Wiry Object Recognition Database - Thousands of images of a cart, ladder, stool, bicycle, chairs, and cluttered scenes with ground truth labelings of edges and regions.
143.	World Bank Open Data	World Bank Open Data
144.	Yale Face Database - 165 images	Yale Face Database - 165 images (15 individuals) with different lighting, expression, and occlusion configurations.
145.	Yale Face Database B - 5760	Yale Face Database B - 5760 single light source images of 10 subjects each seen under 576 viewing conditions (9 poses x 64 illumination conditions). (Formats: PGM)
146.	Yelp.com Datasets Challenge	Yelp Dataset Challenge: Yelp reviews, business attributes, users, and more from 10 cities
147.	YouTube-8M Dataset	YouTube-8M Dataset - YouTube-8M is a large-scale labeled video dataset that consists of 8 million YouTube video IDs and associated labels from a diverse vocabulary of 4800 visual entities.
148.	Aylien News Data API
149.	Aylien Datasets
150.	Finance Datasets on Kaggle
151.	Github philipperemy/financial-news-dataset
152.	FT Markets Data
153.	IMF Datasets
154.	Worldbank Datasets
155.	Reuter Finance News Dataset Title Only
{156.	Bloomberg + Reuter Finance News Dataset

Conclusion:

Machine learning datasets play a pivotal role in the development and advancement of various machine learning applications. In this article, we have explored an extensive collection of datasets obtained from more than 150 data sources, encompassing classical machine learning, computer vision, NLP/NLU, audio processing, and time series analysis.

By leveraging these diverse datasets, researchers and practitioners can build more robust and accurate machine learning models. These datasets provide the necessary ingredients for training, testing, and validating models across different domains, enabling the development of intelligent systems that can understand, interpret, and make predictions from complex data.

As the field of machine learning continues to evolve, the availability of high-quality datasets remains crucial. Whether you are embarking on a new project or seeking to enhance your existing models, exploring and utilizing these curated datasets will empower you to push the boundaries of what is possible in machine learning.

Remember, the power of machine learning lies not only in the algorithms and techniques but also in the data that fuels them. Embrace the vast array of datasets at your disposal and embark on exciting journeys of discovery and innovation in the world of machine learning.