Kind of like last year's Search-Script-Scrape, except more open-ended and more demanding...but, with more context. The main purpose of these challenges is to not just continue honing your programming skills, but to acquaint you with the many datasets, and their many possibilities and limitations.
I'll add more context to this page, but basically, every week, you'll have to pick several problems to work on. You can work on them with a partner. However, do not team up as a class. And do not pick the same topics, over and over. Your ignorance of data is going to be the main limitation of your project work.
Tentative schedule
|
Tier 1 |
Tier 2 |
Tier 3+ |
Week 2 |
2 |
|
|
Week 3 |
3 |
|
|
Week 4 |
|
1 |
|
Week 5 |
2 |
1 |
|
Week 6 |
1 |
1 |
|
Week 7 |
|
|
1 |
Note: You can choose to do a single Tier 2 exercise rather than 2 Tier 1 exercises. Or a single Tier 3 exercise just for the fun of it. You don't get extra credit though. And you can't trade downwards, i.e. doing 2 T1 exercises instead of a single T2.
Tier challenge levels
Tier 1
These are meant to be one-off exercises that can generally be done in a single script. And the dataset to fetch and analyze is generally pretty small so that you don't have to worry too much about memory or disk space or bandwidth constraints. The output may be a chart or a few lines of text.
Tier 2
Think of it as a Tier 1 task, except made scalable. Instead of counting the number of crimes per precinct in a given year, you have to fetch a decade's worth of data, collate and wrangle it, and create 10 charts instead of just 1 – or at least, design a chart that can convey 10 times the data. You might also have to work with multiple datasets and APIs and find something interesting in their intersection.
Tier 3
These aren't harder than Tier 2 in a software engineering sense. But the answers are more open-ended and involve more explanation and thoughtfulness on your part. You'll have to be confident in being able to compartmentalize and organize your information…because you might have a lot of it in some of these challenges. Some of these challenges are also an opportunity to try more advanced tools and libraries, such as for statistical learning, that we don't really focus on for the core curriculum.
Tier 4
I don't know if all of these are necessarily harder than Tier 3 challenges. It's just that I generally have a good idea of how to efficiently solve all of the Tier 3 challenges. But the Tier 4 challenges have various unknown unknowns – or at least some kind of quantification that cannot be solved through math alone. So you might end up doing as much research and reporting as actual programming to figure out the right heuristics. Most of these probably aren't (satisfactorily) doable in a week.
Tier 1
- List U.S. states in order of median age of their Congressional represenatives
- List U.S. states that currently do not have a woman in its Congressional delegation
- List the 5 most popular baby names for each U.S. state for the year 2014
- Map all California kindergarteners in which 15 percent of the class have personal belief exemptions against being vaccinated
- The 50 most common words in Shakespeare not found in the 50 most common words in the King James Bible
- Before Justice Antonin Scalia's death, the last time that Justice Clarence Thomas spoke during oral arguments.
- The 20 most common trigrams between Shakespeare and the King James Bible
- List the number of Congressmembers who have not filed their most recent annual personal financial disclosure form electronically
- List all current Congressmembers who have yet to turn in last year's personal financial disclosure form
- Computationally determine the real Donald Trump campaign committees among the FEC data
- How much have independent expenditure-only committees spent to fight Trump versus Cruz?
- Create a monthly time-series chart showing the amount of independent expenditures spent for/vs Hilary Clinton
- List/map the 10 biggest earthquakes worldwide for each year in the past 10 years
- List/map every CAT3+ Hurricane in the last 10 years
- Count the number of road accidents in the U.K. that are suspected to have been influenced by alcohol
- Count the number of fixed red-light cameras currently operating in Chicago
- Graph a time-series showing the diversity of baby names by year for the United States
- Which U.S. states are the least diverse when it comes to diversity of baby names?
- Which U.S. states are the least diverse ethnically according to U.S. Census numbers?
- Chart a time-series of NYPD stop-and-frisk activity by month and year
- Chart a time-series of NYPD reported felonies by month and year
- Chart the annual number of U.S. police officers who have been killed in the line of duty since 1950
- Chart the annual number of officer-involved shootings for the Dallas Police Department
- List the most active U.S. State Department Twitter accounts
- List the most popular Instagram accounts operated by a federal U.S. agency
- Number of Clinton emails that have yet to be fully declassified
- The lowest-level of education you can have and still be a U.S. astronaut
- The number of astronauts who have military experience
- List U.S. senators by percentage of votes missed
- The average health inspection rating for the most popular restaurant franchies in San Francisco
- The nursing homes with the most accumulated federal penalties and fines in the latest year
- A list of motor accidents in New York with at least one fatality
- Highest paid job titles in the White House
- Collect all FOIA requests filed with the city of Chicago
- Calculate in-custody deaths by year at California's largest jails
- Map all known U.S. public and private prisons
- Late-filed amendments to Congressional personal finance reports
- Most commented federal rules that are up for commenting
- All cancer trials that are currently recruiting for patients in California
- Everyone who who works at Goldman Sachs who has donated $200+ in a federal campaign this cycle
- How many days until the next execution in Texas?
- The month in which the most inmates were executed nationwide in modern U.S. history
- Amount of general credit card spending in Congressional quarterly reports
- How many federal websites are currently down/broken?
- Which NYPD stop-and-frisk report corresponds to Giancarlo Esposito's experience
- The most far-flung places visited by former Congressmember Aaron Schock
- The 20 artists most similar to Beyoncé in perceived musical style
- Map the mass layoffs in the state of California
- List all finished U.S. clinical trials that have not yet released their results
- The nearest asteroid to us, right now
- Number of recalls on products that have killed children
- Number of Donald Trump-related tweets in 10 minutes
- How many Dallas police officers have been involved in multiple shooting incidents?
- Is the L Train fucked?
- For a given river point, how good is the flow compared to a year ago?
- Total number of people whose health care information may have been compromised in a mass leak
- Every reported missing child in the U.K. in the last six months
- How many people are using Internet Explorer to view a federal government website right now?
- The most active 18F Github repositories
- From @realDonaldTrump's tweet stream, separate the retweets from his actual tweets
- When did the official White House photographer get a new camera?
Tier 2
- Number of times ISIS/ISIL/Al-Qaeda has been referred to, by year-month, in a White House press conference
- All New York restaurants that have at least one rodent-related health-inspection violation in the last year
- Calculate percentatge of Senate votes missed based on month and releative time distance from general election
- The alma maters of every current congressmember
- List all Dallas PD officer-involved-shootings in which the officer had 2 or fewer years of experience
- Collect and collate all earthquakes that have occurred within Oklahoma's borders or impacted its cities since 2008
- Graph how diversity in baby names correlates to ethnic diversity (as the latter is measured by the Census)
- List the most important U.S. product recalls
- List/map every CAT3+ hurricane that was CAT3+ when it made landfall
- List U.S. states that have never been represented by a female Congressmember
- List U.S. states in order of their Congressional delegation's average number of years served
- Which Congressmembers are most likely to write their own tweets?
- How many current Congressmembers were not born in the United States?
- How many current Congressmembers represent a state that they were not born in?
- For each state, list the top 10 most popular baby names in 2014 that are not among the top 50 baby names in any other state
- For the past 10 years, find the 10 most deadly earthquakes in terms potential human casualties, worldwide
- List all former Congressmembers and senior congressional staff that are currently eligible to be lobbyists
- Collect and wrangle all of the White House visitor records into one database
- The biggest reward currently offered by the FBI for wanted bank robbers
- The most active/popular Twitter account among NYPD precincts
- List the 10 musical artists most like Johnny Cash but whose careers began after his ended
- List the baby names common to the U.S. and the U.K. that have the greatest gender difference (e.g. Robin)
- Collect and collate the meetings FDA officials have had with persons external to the executive branch
- California kindergartens that have changed the most in immunization rates and personal belief exemptions
- Chart the correlation between change in stop-and-frisk activity and reported felonies, by NYPD precinct, for the years 2011 and 2014
- Chart the ratio of NYPD stop-and-frisks that resulted in an arrest, by racial group, by year
- Computationally determine the Twitter accounts for all currently active New York Times reporters
- Collect and collate the FBI Uniform Crime Report spreadsheets for each city
- Count the number of cities that have reported a drop in burglaries but not in vehicle thefts
- Which news publication has grown in popularity, according to paid Congressional subscriptions?
- The biggest police departments to have reported the fewest hate crimes in 2014
- The ratio of businesses penalized for major workplace safety infractions over the years
- Airline routes that exist at San Jose airport in March 2016 that did not exist in March 2009
- Workplace accidents involving non-fatal falls of 5+ stories
- Funniest Supreme Court justices, ranked in order of number of times people laughed while a Supreme Court Justice spoke during oral argument,
- September Elevenths, ranked in least amont of front page space in New York Times, by year since 2011
- New York Times feature obituaries by gender, since 2010
- Nursing homes with severe violations in consecutive inspections
- A list of intersections and street sections that have had multiple fatalities in New York
- Average time, in half-hour increments, that it takes to get from Wall Street to JFK on weekdays
- All counties that switched from Republican to Democrat or vice versa in terms of voting for president, 2012 vs 2008
- Every vehicular motor crash that occurred on a Friday or Saturday night
- The largest datasets in terms of rows x columns for a given city's open data aportal
- Creatiing a U.S. census crosswalk between redrawn Congressional districts between 2000-2010 and 2010-2016
- The most gentrified part of San Francisco, according to change in crime patterns
- The most gentrified part of San Francisco, according to change in income measures by Census
- Find all FOIA requests filed for Chicago in which the requester is a journalist
- Calculate the number of days it took for the Washington Post to complete information for each police shooting victim
- Find all the police shooting victims mentioned in the Guardian's count not mentioned in the WP's, and vice versa
- Collect and collate all the bulk detailed data sets from FEC
- Every inmate executed by Texas who did not mention God or religion in any fashion in their last words.
- BuzzFeed's favorite animals
- Types of crime categories with biggest change in reports over time
- How many URLs as cited by the Supreme Court are currently down/broken?
- List all photos in the Flickr White House feed consisting of a group of 15 or more people
- List the most recent Instagram selfie for each Congressmember
- Computationally determine the most competitive Congressional races using FEC data
- Collect and wrangle all of the New York Police Department stop-and-frisk records into one database
- The most interesting words or phrases in each of President Obama's State of the Union addresses
- List the departments by biggest change, positive and negative, in terms of number of sworn officers between 2008 and 2014
- Which U.S. healthcare companies have the best record in reporting their clinical trial results?
- Aggregate the number of health care data breaches related to paper vs. onsite computers vs remote hacks
- Most followed non-political Twitter celebrities in terms of bipartisan Congressional followers
- Most partisan celebrities, as determined by partisanship of foillowers who are Congressmembers
- Richest congressmember who represents poorest distrct (or vice versa)
- Find all free lunches being served at Stanford events
- Rank Stanford calendar events by level of general interest
- Which Congressional races are receiving a majority of their funding from outside the state?
- Political ad spending on Google/Twitter/Facebook by election cycle
- Most popular categories of federal jobs that are being hired for
- Rate of tweeting per Congressmember
- Best workout songs on the Billboard Top 100
- For a given Congressmember, all votes in which they voted against their own party
- Time series of Wikipedia page views for each presidential candidate
- Rank the most popular New York subway stations by turnstile traffic
- Find all congressional votes that happened late at night or over the weekend
- The 10 most popular medical specialties
- Map every city in the world with at least 15,000 people
- Is this the least productive Congress?
- Find every official UK Police Twitter account
- Chart the history of Stanford's tuition hikes and include inflation
- Winners of Nobel Prizes, sorted from youngest to oldest, relative to when they received the prize.
- Commits made to goverment open-source repos during the weekend or during off-hours
- Longest applause during President Obama's States of the Union
- The U.S. county with the most Starucks per capita
Tier 3
- The poshest New York restaurants to have ever received a C grade or lower from the health inspection department
- Which least black Congressional districts are currently represented by a black Congressmember?
- Chart the correlation between change in white-nonwhite population, and Republican vs. Democratic share of vote in 2004 vs 2014
- List all former Congressmembers and senior congressional staff that are currently eligible to be lobbyists but have never registered as lobbyists
- List all photos in the the White House Flickr stream that do not show President Obama's face
- Computationally create a list of every official police department-operated Twitter
- The most interesting datasets on data.gov
- The power of Congressional committees as determined by campaign finance records and tenure
- The most astroturfed federal regulations
- The most popular alma maters among all U.S. Congressmembers
- Number of memorable quotes as given by gender of character
- Most scene-stealing actors
- Number of pieces reported by BuzzFeed staff versus BuzzFeed community
- Analysis of printed word count by section over the years for the New York Times
- Analysis of what makes a book more successful on NPR's bestsellers list compared to NYT's
- Which new doctors did Eli Lilly hire as speakers/consultants in 2014 compared to 2011?
- Average education level of the Texas death row inmate
- Analysis of city-level misdemenaor and low-level drug arrests after marijuana legalization laws passed
- Highest poverty schools with the highest 5-year improvement in federal testing benchmarks
- All Congressmembers who have served in the military
- Partisanship measure for news media, based on partisanship of Twitter followers ane engagement
- Counting number and kinds of NYT stories that have at least one anonymous source
- Who among the Guardian's columnists writes most like a New York Times columnist?
- Map Starbucks locations with median income of U.S. census block
- Rank and correlate gentrification of New York neighborhoods by Census demographics and Zillow price data
- Is there a correlation between wealth of a neighborhood and response time when it comes to 311 reports, when type of report/agency is accounted for?
- Simplify taxonomy of Jeopardy categories
- Public schools near environmental hazards
- Rate of press releases per Congressmember
- The most important Congressional votes
- Do disciplined Chicago police follow the bad apples theory?
- Which California cities and universities reserve the highest portions of their payroll for administrators?
- Analysis of Wikipedia page size/number of edits/editors by gender of subject (e.g. authors, actors, scientists, movie characters)
- Media coverage (NYT/NPR) of the major presidential candidates
- Scrape and parse and collate historical expenditures on endangered species
- Most expensive drugs and which medical specialties handle them
- Analysis of home financing data and Census demographics
- Comparison of pollutant levels at individual sensor levels
- Find unique, useful adverse event reports from the FDA FAERS
- Where are the stationary traffic cameras in New York?
- DrudgeReport's favorite news sources, over the years
- Diversity of police department compared to diversity of jurisdiction
- For every school in America, find the nearest M3.0+ earthquake that took place during school hours.
- About how many Instagram-using tourists were on Stanford's campus yesterday
- Every federal campaign contribution by someone who has worked in Hollywood and is somewhat famous
- Replicate OpenSecrets' "Sex, Money, and Power" Analysis for 2014
- List the biggest Congressional upsets for each federal election cycle
- Given a photo of a prescription pill, list its generics
Tier 4
- Ethnic/racial demographics of U.S. Congress, historically
- The most interesting ways language has changed between press releases authored by the Bush and Obama era Justice Departments
- Congressional issue lobbying that seemed to have the least effect
- Which fields of medicine are Eli Lilly and Pfizer most interested in?
- Analysis of recidivism based on Florida inmate data
- Suspicious car defects and complaints that haven't yet resulted in an investigation or recall
- How has OSHA changed the way it inspects for potentially fatal health hazards?
- Detection of particularly dangerous consumer products according to hospital injury data
- Possible conflicts of interest between Supreme Court members' personal finances and cases they preside over