WebDriver For Journalists: Scraping the Web To Report the Truth

Did you know that in 2021, a Pulitzer Prize was awarded to a project that had WebDriver code at its core?

The New York Times COVID data-tracking project became the United States' most-watched dashboard for tracking changes in the spread of the pandemic. It worked by aggregating data from municipalities across the nation. These sources ranged in sophistication from sanitized data available for download, to bespoke HTML maps.

In this presentation, we'll discuss the role of WebDriver and other web-crawling technologies in that and other journalistic endeavours. 
We'll review using selectors to find data for journalists, cleaning source data, and the value of agility in deadline-driven workflows. 
We'll also explore how the lessons learned in this line of work are applicable to the practice of software testing and beyond.