Rob Frydlewicz Observes and Reports the Weather … Using Big Data

It is so hot this week that my thoughts naturally turn to the weather. A recent article in CIO Magazine titled How to Profit From the Ultimate Big Data Source: The Weather inspired me to talk with former media industry research executive Rob Frydlewicz about analyzing  weather using Big Data. Rob publishes the NYC Weather Archive blog and for years has been examining large data sets for strategic insights and forecasting purposes.

In talking to Rob about how he mines this data and divines his insights, I realized that the true value of a strategic researcher is their ability to discern instructive patterns from seemingly disparate numbers and make accurate valuable predictions from them. It is arguably one of the most important talents needed in today’s business world. 

Rob explains how he uses weather data in these video:

CW: As a media researcher, you have always been surrounded by lots of data. What makes today’s data different from years ago?

RF: A generation ago, a stack of computer printouts dumped on your desk was referred to, usually disparagingly, as a “data dump.”  Since then these “dumps” have grown exponentially and are now handled with reverence and referred to glowingly as Big Data.   They are all around us, with most attention going to respondent-level TV viewing data, Wall Street’s second-by-second transfer of millions of dollars and shares, and the granular telephone and online usage patterns being turned over to the NSA. 

CW: Rob, tell me about your use of weather data for your blog. How did you get started in it?

RF: As a media research professional I’ve had plenty of experience being immersed in media usage, advertising spending and Census data, to name just a few.  And recently I added weather data to my repertoire.  In my weather blog, NYC Weather Archive, I tell stories about interesting atmospheric conditions  that have occurred in New York.  The building blocks for these analyses are the detailed, daily weather information collected by the National Weather Service.  When I began the blog, its scope was limited to the data I subscribed to.  Then, last year, all of its weather data, for as long as it’s been collected, was made available free of charge.  This greatly enlarged the possibilities for further interesting analyses.

CW: With so much raw data, how are you able to make sense of it? How do you even start to analyze it?

RF: It’s only by wading through the data that many of my posts have taken shape.  Very often, as I pore over the statistics with a particular story in mind, other patterns form before my eyes and I find myself with other stories to investigate.  But unlike so much other person-based Big Data, no one’s privacy is compromised by looking through all of this weather data!

CW: What is your advice to companies seeking an expert in Big Data analytics?

RF: Big Data is only as powerful as the persons who sift through it - analysts who thoroughly enjoy probing the data, discerning patterns and shaping fascinating and useful stories. 

CW: Can you offer a list of blog posts that have used Big Data and have resulted in interesting insights?

RF: Yes. Some of my favorites include – March Madness – 80-Degree Weather in March; When the Rain is Unrelenting; Biggest & Smallest Temperature Swings; Cold Weather, no Snow – A Snowlover’s Nightmare; How Common Are “Average” Temperatures?”; Too Cold to Snow? NYC’s Biggest Snowfalls During Frigid Weather; Unbearably Warm Summer Nights.