While the data was being collected, I had formed some notions about what kind of analysis I would perform and how I would present the analysis. I wanted to restrict myself mostly to frequency and statistical analysis that would give me some idea of the distribution of the tweets and present these in a visually appealing form. I broke my analysis into three distinct areas:

• Analysis of the tweets and their content
• Geographical analysis of the tweets
• Analysis of the tweets vis-à-vis the characters of the TV series.

For each of these areas, I worked out some data aggregation and analysis that I could perform using a combination of SQL, Apache Pig + HBase and Ruby. While a lot of the analysis could have been done using standard SQL (after all 2.72 million isn’t exactly “Big Data”), I chose to use some of the technologies that are increasingly being associated with Big Data, just to get an understanding of the technology.

I finally visualized all that data using some nifty visualizations from D3.js to get a better feel for the data. The data can be seen in the companion micro-site and has been presented in a prior blog post.
The Summary

Summary of Tweets

These were just summary aggregates generated using MySQL scripts. I used regular expressions to detect links while I was loading the data into the MySQL table.

Tweets over Time

Timeline of Tweets

Game of Thrones aired on HBO on every Sunday night. Overall the Twitter chatter peaked on the day the episode aired and the day following the air date. Seeing a different view of the timeline actually reveals a bit more insight into this observation.

Tweet volume by Day of Week and Hour

The heat map above shows the distribution of tweets over hour of the day and day of the week. All the tweets have been normalized to the EDT Time Zone so that the analysis could be put into perspective with the Time Zone in which the TV series first aired. In line with the air date and time, we see that busiest hours seem to be on Sunday immediately after the week’s episode airs in the US. A second “peak” is seen on Monday in the two hours after the episode airs in the UK. The rest of the world catches up over the week, with Saturday being a lax day. Finally the anticipation builds up again on Sunday.
The visualizations used here were built using D3.JS. NVD3.js, a D3.js based library provided the visualization for the stacked area graph. The heat map was “inspired” from this example in the D3.js example code gallery. The data was aggregated using SQL and Ruby scripts. Ruby scripts were also used to generate the JSON data needed for the visualizations.

What were people saying

What’s equally interesting is what people were saying about the show in their tweets. I was very interested in what words and by extension “hashtags” were used by people. To help understand this, I did a word frequency analysis on all the tweets using HBase and Pig and visualized it as word clouds (using D3.js) for both “hashtags” and words.

Word Cloud of Tweets

Clearly certain words stood out and underscored reactions of people to the last three episodes. Hash Tags clearly showed the three episodes viz. “The Mountain And the Viper”, “The Watchers on the Wall” and “The Children”. Other show themes like “Trial By Combat”, “Valar Morghulis” and show star characters like “Tyrion”, “Arya”, “The Mountain”, “The Viper” etc. also dominated.

A word cloud of Hash Tags

Both the word clouds were based on Jason Davies implementation of the word cloud using D3.js. This library is a fairly nifty one to visualize word clouds, but I haven’t been able to fully customize it as needed.

Who was popular

Popular Characters

Another thing to check was what characters people were tweeting about. Here I tried to once again do a word cloud, but focussing just on the characters. Five characters clearly stood out viz. Oberyn Martell (The Red Viper), Jon Snow, Arya Stark, Gregor Clegane (The Mountain) and everyone’s favourite Tyrion Lannister.

Characters by Tweet volume and Day

To get a more qualitative feel of character mentions, I tried to look at volume of mentions per character. Here too some clear trends stood out:

• The final episode very neatly tied up the character story arcs for most of the many characters in this show. Naturally, a significant number of characters saw a high number of mentions in the last episode.
• Ygritte and Jon Snow, both saw the highest number of mentions in the episode “The Watchers on the Wall”, since that episode almost exclusively focused on their part of the Game of Thrones universe
• Oberyn Martell and The Mountain, naturally saw the maximum number of mentions in the episode “The Mountain and The Viper”, which featured their epic showdown

The character cloud was once again based on Jason Davies word cloud implementation. The character “tweet count by day” chart was based on another example from the D3.js examples page called “Publications in Journals over Time” by Asif Rahman.

Where were people tweeting from

What was equally interesting was where people were tweeting from. Unfortunately, location data is not often available in tweets and in this case a mere 3-4% of tweets had geo-location data. Nevertheless, even this amounted to a fairly decent number of data points. I took this data and I did a kind of a “choropleth” world map that is shaded by number of tweets. This map was based on the “datamaps” library by Mark DiMarco and is an excellent library for these kind of mapping visualizations.

Countries by Tweet Volume

The first thing that I noticed is the sheer popularity of this show. Tweets had poured in from all over the globe, including those that appeared to be gibberish from a ship!!!

The overall distribution of tweets was also indicative of the fact that the show was immensely popular in the US and UK, which was something I noticed in the timeline data as well.

Tweets by Location

I also plotted all the points on an interactive Google Map that you can see here. I used CSS to style the map to that wonderful grey color that went well with the rest of the visualizations. Initially, I tried loading all the data points using a GeoJSON layer, but that proved to be very slow. So sacrificing some control over the look, I eventually opted to use Fusion Tables with Google Maps for this.

So What Next?

So far I have had an interesting month learning all this new stuff, and also playing around with data visualization. One of the things I missed here was actually storing the tweets in a database or in as system designed to handle the kind of volumes that would be seen in a real life use case. So as a next step, I plan on learning and moving to Mongo DB or some other similar data store that would serve me better than just using a flat file.

Also, a lot of the analysis that was presented here was done in a semi-manual fashion. I’d really want to automate this entire process so that it becomes seamless from data collection to visualization. As an end goal would ideally love to see this evolve as platform to solve some domain specific data visualization problems. I’ll keep updating this blog as and when I make progress on this, so do watch this space for more.

I had been searching for good topics to collect data on, when I had the notion that I could use my current favorite TV Series – Game of Thrones – as a kind of test bed for this.

Ever since I watched the Red Wedding episode on Game of Thrones, I have been kind of determined to not let the show pull another fast one on me. In fact, soon after the end of that season, I actually read all the books in the series, just so that I could stay ahead of the TV series. Thanks to this, I knew what was in store for the final three episodes of Season 04 and figured that this would make for a good experiment on Twitter.

Luckily for me (or rather unluckily) the show was on a short break due to the Memorial Day weekend, when I had the idea. That meant that I had less than a week to get started and begin the data collection before the show resumed airing on HBO. What followed was a rather interesting ride through technology, visualization and data insights.

Gearing up for D-Day

So having decided to do this, I began by trying to get a good handle on the technologies involved. Luckily, support for the Twitter API is quite good with a number of libraries readily available for a variety of programming languages. Having previously done similar analysis using Ruby, I chose to use the Twitter gem for Ruby. Implementing the Twitter Streaming API using the gem was quite straightforward and in a few evenings (yes, I have a day job) I had a good enough system working on my development environment. Once I had access to the data I was faced with the next decision – should I write this to a database or a flat file?

I originally intended to use Mongo DB, but given the steep learning curve in accessing and processing the written data,  I finally chose to go with the flat file approach. Also I chose to only write a few fields viz. Tweet Creation date, Tweet, Twitter Handle, User Name, Retweet Flag, Location, Time Zone, Latitude and Longitude. In hindsight, I probably should have included a lot more fields but more on that later.

Given the volumes I had seen during my tests I presumed that over a 3 week period I would probably end up getting about 700K to a 1 million tweets. So given those kind of volumes, I figured I’d just host the data collection script on the server that hosts my blog. That’s where I hit the first roadblock – the ruby environment there was just plain out of sync with reality, running some really old versions of gems.

I realized this with just two days to go and at the very last minute decided to try and host this on AWS. Doing this on AWS was a breeze and in one long night, the code was finally live!!!

The Data Avalanche

I had written the data collection script to generate a new file with tweet data for each day. The file generated for June 01st 2014 was well within my estimates in terms of number of tweets collected, but nothing prepared me for the avalanche of tweets that just poured out when the monumental episode “The Mountain and The Viper” aired. That day generated close to 500K tweets, with the total for the week crossing the 1 million mark, which itself was as much as I had estimated for the entire three week period.

Surprisingly, the script didn’t crash at all and was still running at the end of the day. In fact in the entire 21 day period, the script only crashed once for a few hours – which given the fact that I had just hacked together the code over a few evenings, was quite satisfying.

Data quality and other woes

But with great volumes of data came great issues of quality of data. I was using a tab-delimited file format to store the data, which at the time seemed like a safe choice. But in spite of that I faced several data quality issues.

One of the common issues I had was with the presence of arbitrary new line characters in the files. While I tried to strip out these characters within the script code itself, some errors still crept through. This amounted to about 0.1% of the total tweets collected.

The second issue was in cases where the text content itself contained tab characters. This was something that I could have easily fixed in the script, but it somehow slipped my mind. This kind of error amounted to about 0.06% of the errors.

I corrected the first error semi-manually, with a script to point out all such errors and the correction done manually. For the second error, rather than find where the error actually was, I chose to just drop those data points since their number was very small.

After correcting all this data, I finally ended up with a clean data file of about 0.5 GB.

Preparing the data and designing the analysis

Overview of data flow from tweet to visualization

So once I had cleaned up the base data, the next step was to figure out what to do with it. The base data file was in a fairly accessible format and for some of the analysis that I intended to do; I could just use that with Ruby directly. For other kinds of analysis, it was imperative that I be able to perform some transformations on the data.

To do this, I chose to use Talend as the data transformation platform, largely because it was open source and easily available. For the analysis I wanted a data set just containing the tweets (for Apache Pig and HBase), another data set with just latitude and longitude (for geographical analysis) and the entire dataset loaded into a MySQL database. Using Talend I built ETL Jobs for each of the tasks and Talend was able to handle this quite easily.

I had originally planned on reverse geocoding the latitude and longitude pairs to extract useful information like the country from where the tweet was sent. After reading up on this, I figured that using a reverse geocoding API like that provided by Google Maps or GeoNames would have been ideal. However, it soon became evident that, thanks to the throttle limits on the free version of these API’s, it would have taken me ages to reverse geocode my dataset. For example Google Maps had a limit of 2500 requests per day, while even the more generous GeoNames (30000 requests per day) seemed inadequate given the volumes of data that I had. On the bright side, GeoNames also made its entire database available for free. Using that and some Ruby scripting, I was able to build an offline reverse geocoder that was able to get the reverse geocoding done in about eight hours.

I finally now had a clean enough data set with supplemental data prepared that I could use to begin analysis on.
I’ll detail out the actual analysis that I performed, next week .

I set up a small experiment where I collected tweets about the hit TV series over the last three weeks of the recently concluded Season 04. Then I added some D3.js visualization magic to the data and presto!!! I had some impressive insights into the twitter chatter around the series.
I managed to collect about 2.72 million tweets. A quick analysis on the numbers showed that about 114K unique users were responsible for those tweets sharing about 102K links. About 36% of those tweets were retweets.

Timeline of Tweets

An analysis of the timeline showed regular peaks around episode air dates with the episode “The Mountain and The Viper” garnering most reactions on Twitter.

Tweet volume by Day of Week and Hour

An analysis of the hour vs. the day, on which tweets were sent out, shows that the busiest hours for tweeting were on Sunday night. After that, the next slot of busy hours occurred on Monday evening. This ties in nicely with the fact that those two slots are approximately just after the time when Game of Thrones airs in the USA and UK respectively. The twitter traffic kind of peters out over the rest of the week, and picks up again on Sunday afternoon, in anticipation of the next episode.

So what did twitter users say about the show? Apparently quite a lot as the word cloud below shows.

Word Cloud of Tweets

Some words that stand out include “Season”, “Finale”, “watch” and words like “Oh”, “Killing”, “Shocking”, “OMG”, “Sad” that kind of sum up the jolt that fans got in the episode titled “The Mountain and The Viper”. The most shared status message was this one from show star, Maisie Williams:

LOVE IS AN OPEN DOOR. #Frozen @GameOfThrones I stole this joke #sorraaaay pic.twitter.com/gjjNbnAFDh

— Maisie Williams (@Maisie_Williams) June 9, 2014

Talk about stars, and you come to the characters on the show, that we have all come to love and hate in equal measure. So, who do the Twitter folk like to tweet about?

Popular Characters

Well, five characters clearly stand out in the three weeks viz. Oberyn Martell (The Red Viper), Jon Snow, Arya Stark, Gregor Clegane (The Mountain) and everyone’s favourite Tyrion Lannister. But if you look at each character, which episodes are the character’s defining moments? The below visualization attempted to answer that.

Characters by Tweet volume and Day

Here I plotted the number of tweets by day for each character, with the size of each circle giving relative tweet volumes for a character. A cursory glance showed some nice insights.
In this season, the final episode very neatly tied up the character story arcs for almost all of the many characters in the show. So most of the characters saw maximum volumes around the day when the last episode aired. However some characters differed from the norm.
For example, Ygritte and Jon Snow, both saw the highest number of mentions in the episode “The Watchers on the Wall”, since that episode almost exclusively focussed on their part of the Game of Thrones universe. Similarly, Oberyn Martell and The Mountain, naturally saw the maximum number of mentions in the episode “The Mountain and The Viper”, which featured their epic showdown.

Countries by Tweet Volume

And where did they tweet from? Apparently the show has universal appeal with tweets recorded from all over the globe, including countries where the show does not air. I wonder now how they managed to watch that
Obviously the show is most popular in the USA and the UK, but I was very surprised to see tweets from places like Greenland, Mongolia, Sudan and even from what seemed like a passing ship !!!

Unfortunately, due to the lack of geo-location data in every tweet, the number of actual tweets that had geo-location data amounted to only about 3-4% of the total number of tweets. Nevertheless, plotting the tweets on a world map resulted in the below graphic.

Tweets by Location

For an interactive version of this map and the other graphical analysis, do take a look at the companion micro-site.
In the coming weeks, I’ll be writing about the technology behind all the pretty visualizations and analysis that were detailed in this post. So do watch this blog for more.
Lastly, a big thank you to Mark DiMarco who helped me with his DataMaps library. Mark, without the quick response to my queries I would have struggled for quite a bit to build out that beautiful “Choropleth” world map that you saw above. Thanks a ton again!!!

The results are out and the BJP has gotten itself a historic landslide victory !!! This means that India can have a stable government for the next 05 years and one not hobbled by coalition politics. Hopefully the new government will use the clear mandate given by the people to end the policy paralysis that plagued the last government and make some developmental progress. However this is not the main point of this post.

In my last post on the elections, I looked at search volumes for each of the leading candidates by region and noted that none of the leading candidates evinced any interest from the south and south-east of India. Looking at the map of India after the results (Google Live Election Tracker), seems that this is the only region where BJP has not made a clean sweep or any in-roads at all. So does this mean that search volumes (or the lack of them) may be a good predictor for election results? Food for thought !!!

The election has been largely dominated by three main politicians at the national level viz. BJP’s Narendra Modi, Congress’s Rahul Gandhi and wild horse AAP’s Arvind Kejriwal. So I pondered over what Google Search trends can reveal for these three players over the last 12 months. The first graph which shows Interest over Time from Google Trend’s is shown below:

Interest Over Time

The graph shows a clear “Modi Wave” dominating the web search queries in the last 12 months. In fact, its more like a Tsunami in the last couple of months. Poor Rahul Gandhi and Arvind Kejriwal are far far behind in this quest to remain at the top of the voter’s mind. They however outshine Modi in one or two instances each. In Rahul’s case, seems his interview with Arnab on Times Now was responsible for the spike in search interest that we see towards the end of  January. Arvind Kejriwal on the other hand spikes well above Modi in December when the AAP party formed the Delhi government. There is one other sharp spike in interest for Arvind around the time when AAP vacated power in Delhi.

What is more interesting is the regional interest in these leaders. Google trends luckily also shows the relative interest by state for each of these leaders. Let’s start with Modi whose chart is shown below:

Interest by Region for Modi

Predictably Gujarat dominates as the region from which maximum interest is shown. In this election, Narendra Modi has turned out to be that rare candidate who has shown some measure of a pan India appeal and this is evidenced somewhat by the search interest graph. However, one should note that the interest seems to be more concentrated in North and North-west India.

Now let’s move on to Rahul Gandhi

Interest by Region for Rahul Gandhi

Rahul Gandhi seems to have a bigger search following in the North, particularly in Jharkhand. Overall again, there is a concentration in the North, but it is not as intense as that of Modi. And finally now lets look at Arvind Kejriwal.

Interest by Region for Arvind Kejriwal

Given that the AAP movement found its first success in North India, during the Delhi state election, predictably search volumes for Kejriwal are concentrated in the North. However, surprisingly the leader finds much lower interest in other parts of the country as compared to Modi and Gandhi.

Interestingly all three charts, show a remarkable “lack” of interest in South India and parts of the North-East. Would this lead to yet another “coalition” government propped up by the parties of the south? Only time will tell…

1. Put the images you want to convert in some location. The convert command will take into account alphabetical order into account while creating pages. So keep this mind when you want the pages in a specific order.
2. Fire up terminal from “Applications > Accessories”
3. Navigate to the folder or location with your images so that your working directory is that location
4. Then for some magic use the following:
   convert *.png myPDF.pdf

That’s it !! You should now have a shiny new PDF named myPDF in the location you saved the images in. The original tutorial seems to indicate that this will only work on the Desktop, but rest assured that this works in any location. Sure saved me a ton of time

The Moto G came in a small little white box. I fully expected the box to only contain the phone and an USB cable (seems the US model ships with only this), but I was pleasantly surprised to find that it contained an AC charger and a hands-free kit.
My initial impression of the phone was – Whoa!!! – this is much slimmer and lighter than I expected. It also had a nice feel in the hand with it’s slightly curved back. Since the phone needed a micro-sim it took me a couple of days to get my regular SIM switched to one. I decided to actually use the phone for about a month or so before writing about it. And boy has it been a fabulous experience. Here is a rundown of the hits and misses:
Build Quality
The phone is mostly plastic (and cheap strong plastic at that), but it surprisingly does not come off as cheap. The build quality is actually fabulous and fits nicely in the hand. The back is slightly curved, and lends to a great grip on the phone. The phone back cover snaps open to reveal access to the SIM card slots.Yes, the Moto G in India is a dual SIM phone, which for me is an added benefit. The battery is not user replaceable and there is no memory card expansion slot, so beyond changing the SIM card there is little you can do. Moto G however offers nifty customization options via what it calls Moto Shells (but more on that later). Overall, I was very impressed with the build quality of the phone.
Hardware, Controls and Ports etc
The Moto G runs on a quadcore 1.2 Ghz Cortex A7 processor, with 1 GB RAM. While this is not the most powerful processor out there, it certainly gets the job done for most day to day tasks. The phone comes in 8Gb and 16 Gb variants, with no memory expansion slots. I bought the higher 16 GB variant, which in my opinion is fairly adequate in terms of storage. The Moto G has a 4.5 inch LCD screen (obviously to keep costs low – so no AMOLED here), that is surprisingly good for an LCD screen. At a pixel density of approximately 326 PPI, this is the most vibrant screen I have seen so far. In addition to this the screen is covered by a Gorilla Glass layer, that makes needing an additional scratch guard unnecessary. I have actually carried this in my pocket, with keys and other metal things, in crowded Mumbai trains and come out without a scratch. Lastly, the phone is supposed to be covered with a water repellent layer, so I guess I will know how useful that is when the monsoons hit India.
The phone actually has fairly minimal controls. It features a power button and volume controls on one side. For some reason, Motorola decided to skip on a dedicated camera control, which for me makes taking photos with the phone slightly awkward. It features a standard 3.5 mm jack on the top to plugin a pair of headphones / hands free kit. The bottom has a standard micro USB port for charging / data transfers. The front of the phone features a 1.3 mp secondary camera and a notification LED (yes, it has a nifty notification LED that lights up when you have messages etc) while the back of the phone has the speaker grille, a 5mp primary camera and a flash. All in all, a fairly decent set up.
Software and Day to Day usage
The phone runs what Motorola terms as its version of “near” stock Android. Rather than following other phone manufacturers and putting it’s own UI (aka HTC Sense, Samsung TouchWiz) on the phone, Motorola chose to keep the stock android UI and only make software enhancements where it made sense. I personally like this approach and it certainly has resulted in a far more zippier experience. So far it has been able to handle most apps without a significant lag of any kind, and has been perfect for day to day usage.
During the launch, Motorola promised that the phone would get KitKat, Android’s latest version. A few weeks after the India launch, it actually followed through on this promise and got the phone upgraded to KitKat via an OTA update. Post the upgrade, had a little trouble familiarizing myself with the new dialler and other features in KitKat, but once I did that its been a delight to use.
The new integrated dialer takes some getting used to, but the voice clarity is fabulous. I also found the speaker on this to be great for taking hands-free calls.
Motorola included a FM Radio in the Moto G, something clearly aimed at emerging markets. The bundled Radio app works quite well, but I found the reception on it somewhat inferior to the kind of reception I get on my “dumb” $40 Samsung phone.
Motorola also threw in a few of its custom app’s like the Motorola Migrate, Motorola Assist etc. which are quite handy. Motorola Migrate is supposed to help you migrate your data etc from your existing phone on to the Moto G. While this may be handy to migrate messages and phone logs, for other items like contacts, photos etc. you don’t really need this app. As long as you sync with your Google account, Android anyway does it for you.Motorola Assist is actually somewhat useful, with features that hold your calls when you are in a meeting or at night (the fabled “Do not disturb” mode). Overall, with the features of Android and the goodies Motorola throws in, it’s been a superb experience to use the phone.
Camera
The Moto G features a 5mp primary camera with a flash. While this looks good on paper, the quality of the pictures taken aren’t all that great, especially in low light. In well lit environments though, the camera takes fairly good pictures. The 1.3 mp front facing camera is quite adequate for video calls. My main gripe with the camera is actually the camera app. With the absence of a dedicated camera button, I have found taking pictures slightly cumbersome, with the entire “touch to click” approach. Barring this, if the camera isn’t the most important thing you look for in a phone, the Moto G provides more than a serviceable option. The gallery below showcases some pictures straight out of the Moto G camera.

Battery
This is where the Moto G shines the most in my opinion. Motorola promised all day battery life and the Moto G certainly delivers. For the last one month or so, I have been using the Moto G with sync turned on all the time and wi-fi on (6-8 hours daily wi-fi usage). I have been averaging about 2 hours of calls each day and using it for maybe about 1-2 hours in aggregate for surfing. And of course applications like Twitter, Facebook and Gmail are on all the time taking in a couple of minutes every hour on the phone. With all this I generally have about 30% or more battery life left on the phone by the time I am back home each day at about 08:00-08:30 in the evening. While my usage is definitely not heavy, I’m sure most people would be able to get by with similar battery usage on a daily basis.

Accessories
As of now the only accessories available in India are the various Moto Shells. You have choice between the colored shells, the flip shells and the grip shells. I didn’t really like the colored shells, and the flip covers weren’t easily available in black. After much thought I went for the grip shell. The grip shell encases the Moto G in a nice rubberized bumper like shell. You have to remove the back cover to fit this on, which I felt was a bit weird. It does fit on quite well, and protects the edges from bumps. Only drawback was that the rubberized front is a dirt magnet, so regular cleaning is needed.
Conclusion
Overall for a phone that costs only INR 14,000, the Moto G is a fabulous value for money buy. If your main need is a reliable phone for everyday use, that you can also use for moderate online usage with your favorite apps – it doesn’t get better than this. Go ahead and buy one today.