So. I want to collect more data. That means I have to add it to the spreadsheet. I also want to enumerate the tweets. Most importantly – I need to be able to handle missing fields in the json I get from Twitter. If there are no hashtags in the tweet, the field “hashtags” wont be there. The script should be able to handle that. A lot of the tweets I have collected, are retweets. Thats fine. But they begin with the string “RT”. Should I remove that?

It is tempting to collect all the data. But I don’t think I’m going to do that. A lot of the fields does not appear in every tweet. And I’m not sure I’ll ever figure out what to do with it. So it’s going to be a selection of them.

Oh, and by the way – why should I break the current spreadsheet? I’ll just add new columns. No need to rework previous graphs. The actual content may vary, but what the heck. And those coordinates that are more or less useless (because no-one uses them)? Lets replace them with the counter.

Anyways which fields?

I think thats gonna be it.

Getting to the hashtags is a bit difficult. First of all, there might be none. Secondly, there might be more than one. Remember that I get the tweet in JSON-format? And I call the variable “data”.

The first hashtag – given that there is one, can be called like this:

data[‘entities’][‘hashtags’][0][‘text’]

The second of course is found simply by replacing “0” with “1”. The third with “2” etc. The number of hashtags can be found like this:

len(data[‘entities’][‘hashtags’])

Putting it together:

try:
for x in range(0, len(data[‘entities’][‘hashtags’])):
hashtag = hashtag + data[‘entities’][‘hashtags’][x][‘text’] + ” ”
except:
hashtag = ” ”
til_regneark.append(hashtag)

Done. I should put some timers in. It does take quite some time from the tweet is caught until it is written to the spreadsheet. Thats simple. Let me just try that.

Yeah. It is writing to the spreadsheet that takes time. About 4 seconds. Not much I can do about that.

And the timing? I already have the “time” library imported. Just place a “start_a = time.clock()” before what you want to time. And an “end_a = time.clock()” right after. And then “print (end_a – start_a)”.

Done. Now I have the data I need. Or I will have in a little while. All that is left is to plot it.

Now we’re getting somewhere! At the bottom of this post will be links to the previous parts of this odyssey.

I now have a spreadsheet with some data. How do I make that interesting to look at? First of all, I have to decide on a framework for graphing this. In part IV, I made a choice. I chose to save the data in a Google Spreadsheet. For two reasons. One is that I know something about how to work with it. As the previous parts have revealed, I dont know much. But I know enough. The other reason is, that Google has a chart API, that I should be able to use. That in itself is not a very good reason. I know how to work with HighCharts, so why not use that, instead of learning something new? Because Google promises, that the graphs will update live, as I fill in more data in the spreadsheet.

How cool is that?

Where to begin? I don’t think that Google actually provides any good examples. A lot of other people does – it just takes at bit of googling to find somthing that is helpful. I think this example is a good beginning. Go on, take a look.

OK, first of all it begins like this:

function drawChart() {
var query = new google.visualization.Query(‘http://docs.google.com/spreadsheet/tq?key=0AhCv9Xu_eRnSdHBELWZoZHRaQTl5VXdRa3JLTzByUlE&gid=1’);

That tells the chart software where to find the data. It does not look like the ordinary link to a spreadsheet. That looks a bit more like this:

https://docs.google.com/spreadsheets/d/14kCXRFEOxBMgbGDhwBP1Zy8yhZhC7ty54iBVlCovKxc/edit#gid=1

I’m not sure exactly what the difference is, but when I follow the URL in the script, my browser downloads a JSON-file. I think it contains the data from the spreadsheet. So I need to take the URL I already have for my spreadsheet, and format it like in the example.

https://docs.google.com/spreadsheet/tq?key=1kDIxZ5jBWYB2eiAjfsM_jRBr2mYWxudORECFApXBULE&gid=3

I’m almost certain that the gid=3 points to sheet number 3 in the spreadsheet. I’ve accessed the spreadsheet from the example, and copied the data to sheet no. 3 in my own sheet. If I change the long string of random letters in the example, nothing should happen.

It did’nt. The graph on that fiddle looks exactly the same. Now, lets try to the real data. That is in sheet number 1.
Lets also change the “query.setQuery(‘SELECT A, B, etc’)” to “query.setQuery(‘SELECT B, A’)”. That should collect the data from colums B and A in the sheet.
Try again. Yep. I’m now succesfully plotting the data.

It does not, however appears to update live…

It appears that Google does not live up to the promises. I’ll need HighCharts for that. If, that is, I can figure out how to get live data.
There is another problem. This is a very bad graph. It plots the timestamp of the tweet, against the coordinates from the tweet (remember that those were allmost always “None”?). Not that interesting. Maybe Google supports some way of counting? Quick test: Exchange ‘SELECT B, A’ with ‘SELECT COUNT(A), A)’. In theory, is that I should get the count of column A on the x-axis, and the timestamp on the y-axis.

Nope. Better take a look at the query language. Its some variant of SQL. The documentation can be found here. Note – the graph above is not the real thing. I might have to change the spreadsheet, and that will change the graph. Therefore it is an image of the graph. The code will be given below.

And now, before continuing, I have to do something about the laundry.

And now, the promised list of the previuos parts:

Part I: The idea, and the hardware
Part II: Getting my hands on tweets
Part III: Deciding what data to save (that is going to change over the next few days)
Part IV: Storing the data
Part V: Fixing some problems
Part VI: Problems that can wait

And the code for the graph:

<script src=”http://www.google.com/jsapi?fake=.js” type=”text/javascript”></script>
<script type=”text/javascript”>// <![CDATA[
function drawChart() { var query = new google.visualization.Query(‘http://docs.google.com/spreadsheet/tq?key=1kDIxZ5jBWYB2eiAjfsM_jRBr2mYWxudORECFApXBULE&gid=1’); query.setQuery(‘SELECT B, A’); query.send(function (response) { if (response.isError()) { alert(‘Error in query: ‘ + response.getMessage() + ‘ ‘ + response.getDetailedMessage()); return; } var data = response.getDataTable(); var chart = new google.visualization.LineChart(document.querySelector(‘#google_test_container’)); chart.draw(data, { chartArea: {width: ‘70%’, height: ‘50%’}, fontName: [“Arial”], colors:[‘#274358′,’#5e87a5′,’#a2cdf6’], curveType: [‘none’], fontSize: [’11’], hAxis: {title: ‘Tweets over time’, titleTextStyle: {italic: false, color: ‘black’, fontSize: 12}}, legend: {position: ‘right’, textStyle: {color: ‘black’, fontSize: 12}}, lineWidth: 2, pointSize: 7, tooltip: {textStyle: {color: ‘Black’}, showColorCode: false} }); }); } google.load(‘visualization’, ‘1’, {packages:[‘corechart’], callback: drawChart});
// ]]></script>
<div id=”google_test_container”></div>

Yep, its messy.

Done. The test at the end of last chapter was a success. An ugly hack to be sure. As long as it works. This is going to be a shorter part in the series. There are a number of different things I should do. But now I’m getting close to the interesting part. A list of stuff that can wait, because I’m impatient:

• I really should have more exception handling. I’m communicating with the internet, and a lot of things can go wrong. During testing I got an Internal Server error from Google. I should be able to handle that, without crashing
• Having handled that, I should be able to collect the tweets that were tweeted during the connectionproblem, and add them to the spreadsheet later.
• I should also be able to handle getting kicked of the connection to Twitter. How to reestablish it?
• There is a potential problem: Twitter does not guarantee that all fields will be be in all tweets. I should harden the script to be able to handle cases where there is data missing from a tweet.
• I know what the hashtag will be. But as I’m going to leave the Raspberry Pi at home, I should be able to handle a situation, where I’m at work at 8 am, the hashtag is announced at 9am, and I have to enter the hashtag to the code, and start collecting.
• Remember the exceptions back in part V? That exception gets thrown each and every time. Of couse the tweets are saved, but I re-authorize the connection to Google every time. Its not important enough to crash anything. But wks should be declared as a global variable. And apparently I can’t figure out how to do that correctly.

That will have to wait. Apart from the last one, that is actully somewhat critical. I’m getting close to being able to visualise some data. I’ll begin playing in a short while, but right now I think I need to spend some time on the couch.

Oh, and for those just joining the party:

Part I: The idea, and the hardware
Part II: Getting my hands on tweets
Part III: Deciding what data to save (that is going to change over the next few days)
Part IV: Storing the data
Part V: Fixing some problems

In part I I covered the basic idea, and the hardware. Now I’m ready to turn my attention on actually getting my hands on some tweets.

Twitter has an API, an Application Program Interface, that allows programs to talk directly with the Twitter-servers. In order to gain access, I need to have an authenticated Twitter account. I already have that, it was not easy, as Twitter requires a phonenumber to be associated with the account. That can only happen if the phone company supports receiving SMS’es from Twitter. My phone company did not until recently.

That hurdle tackled, I go to the Twitter Apps site. I have to be logged in, and click “Create New App”. After I’ve filled out the required information, I’m provided with four long random strings. I’m not going to show them here. Lets just say that they are long, and contains both digits and letters, both upper and lower case. I get a Consumer Key and a Consumer Secret, basically a login to Twitter. And I get an Access Token and an Access Token Secret – that are specific to the given application I’m gonna write. Those four strings are my credentials, that I’m going to use to gain access.

Next step. I could write a program that sends a request to the Twitter API, and receives data. Or rather, I could’nt, because I’m not that good a programmer. As luck would have it, I’m not the first person in this situation.

But I’m getting ahead of myself. The first question should actually be: Which language should I write this in?

The de facto preferred language on the Raspberry Pi is Python. Python is not the fastest language in the world – its an interpreted program rather than a compiled. But it has a very large userbase, the fact that it is interpreted makes development quick. And it is very popular in data science circles. Propably because it is pretty accessible to people that are not really programmers.

I could write a program – script actually – from the bottom up, that handles all the connectivity with Twitter. Or rather I can’t. I’m not that good a programmer. But someone else has already written a library, a collection of code, that handles all the really difficult parts. Python comes with a LOT of libraries. One of the libraries is called Twython, a portmanteau of Twitter and Python. The intelligent reader of this post, can guess what Twython does.

I do not need to understand everything myself. Others have done that, and on this page, I find an example of how to get tweets with Twython in Python. The code is as follows:

from twython import TwythonStreamer

class TweetStreamer(TwythonStreamer):
    def on_success(self, data):
        if 'text' in data:
            print data['text'].encode('utf-8')

    def on_error(self, status_code, data):
        print status_code
        self.disconnect()
# replace these with the details from your Twitter Application
consumer_key = ''
consumer_secret = ''
access_token = ''
access_token_secret = ''

streamer = TweetStreamer(consumer_key, consumer_secret,
                         access_token, access_token_secret)

streamer.statuses.filter(track = 'python')

OK. I enter my credentials from Twitter, and runs the code. sudo python collect.py.

And tweets containing the word “python” is streaming down my screen:

Actually it was a bit more difficult. The Twitter API requires a safe connection, and none of the requisite libraries handling safe connections were installed. That took quite a lot of googling. A pro-tip: Read the error messages you get. Some of them contains links to solutions.

I must admit that I’m not quite sure what is actually happening. Object-based programming is not my strong suit. Never mind, I just need to understand enough.

The line

streamer.statuses.filter(track = 'python')

Tells the API that I’m interested in tweets containing the word “python”

And this part of the script:

    def on_success(self, data):
        if 'text' in data:
            print data['text'].encode('utf-8')

Does stuff to the tweet I receive. The tweet is returned from Twitter in a format called JSON. And it is saved in the variable called “data”. There are different fields in the format, one of them is “text”. I can access the content of that field by calling data[‘text’]. I may need to do something about the encoding. People tweet in a lot of different languages, so tweets are encoded in the character set utf-8. As you can see in the screenshot, my computer does not really know what to do with exotic characters.

There are a lot of other fields containing data in the tweets. But the main objective is achieved – I can get tweets on my screen in realtime. I you tweet something containing the word “python”, I will get it on my screen moments after.

The second part of the mission is accomplished. On to the third. But first – we’re going to the mall for a burger.

Now we’re getting somewhere! It took most of last night reading up on long forgotten SQL-skills. I present to you, the most prolific tweeters:

OK, I need to do something about the graphical stuff. The text is pretty cramped.

The magic? This SQL-style selection.

SELECT C, COUNT(A) GROUP BY C ORDER BY COUNT(A) DESC LIMIT 15

Combined with choosing BarChart rather than lines. It really is that simple. Although it certainly was anything but simple to figure out (until I gave up, and consulted my sql-books).

What else? I want a graph showing the number of tweets as a function of time. I would also like a wordcloud. And a tagcloud. I dont think I can make that with the google framework. Or maybe I can: I’d better take a look at this page. The tag-cloud would be easier if I import other fields from the tweets to the spreadsheet. 

And if I want a count – it appears that I have to add “=ROW()” to a column in the spreadsheet. That will, I hope, give me a counter.

Of course, that is going to seriously mess up the chart above. But it will only take a change of two letters, to fix. Next step will be to import all fields from the tweets to the database. Just a moment 🙂

Data is the next big thing. Big data is even bigger. And as always when something new happens, libraries jump on the bandwagon from the beginning.

We are coming to it at bit slow this time, at least in Denmark. We might have been intimidated by the Big in big data. But we’re getting there. And as always when we do something new, we have the problem of figuring out exactly what we should be able to do. Wich competencies should we have, what is the level of support that we should provide to our patrons?

I imagine that other places trying to get into data science, would begin by hiring a data scientist. Or at least send the most qualified personel on courses and training sessions. Not so in a library. We have all the books, we’ll just read them, and then we will know how to do it. It actually appeals to me as an engineer:

BOSS: We should do some data science
ME: OK, do we have anyone on staff that knows anything about that?
BOSS: No.
ME: Hmmm. Are we going to hire someone who knows anything about data science?
BOSS: No.
ME: Are we going to invest in training or courses to learn about it?
BOSS: No.
ME: OK, lets get started.

It gives us a chance to play around with exciting and difficult stuff, without being burdened with any actural knowledge about what we are doing. That is more or less how engineers define fun!

So. What should we be able to do?

1. We are not going to be able to advise our patrons about what statistical test to use. There are simply to many. But we should be able to ask qualified questions: “Are you sure that that data follows a normal distribution?”
2. We are not going to be able to advise our patrons about what data, and what correlations are interesting. We have no idea whether a correlation between a biomarker and a disease is relevant. But we should be able to recognise a correlation. Take a look at this page – its hilarious.
3. We are not going to know all the intricacies of all the software we provide access to. Theres a reason we have 79 books in our catalogue about ArcGis. But we should be able to perform a simple data analysis in each of them.
4. And of course – we are going to provide the world class level of customer service and assistance, that make libraries famous. Now we’re just going to assist with data science.

Thats it. How are we going to get there? I’ll get back to that later – now I have a meeting on LibQual.

Note: I’ll be using the male pronoun. Not because there are only men in the world. But because I really like the association between Man-Month and Mothman. For a long time, I thought that the title was inspired by the urban legend of the Mothman. It probably isn’t. I still like it.

In a previous post I discussed the Bermuda Triangle of Project Management. Or the Iron Triangle as it is properly known. I promised to return to the mythical man-month later. Lets do that.

The following thoughts are not my own. They come from a classic essay by Frederick Brooks, written in 1975. And it appears to have been completely forgotten. Well. Not forgotten, it is a classic. But apparently no-one has learned anything from it.

Think back to the ideal world of project management. In that world everyone knows, that if we cut ressources, the project will be delayed. It follows logically, that if the project gets delayed, we should assign more ressources to it. That might be a good idea. It might be a horrible idea. And in some situations, the thought of getting an extra man-month should fill the project manager with the same terror as the mothman did to the good citizens of Point Pleasant back in 1966.

The basic idea is, that if the project runs late, it should get more ressources, in order to finish on time. The observation is, that the added man-months will actually delay the project even more. Why would that happen?

First of all, in most projects, men and months are not interchangeable. We all know that some of our colleagues are more productive that others. Maybe they just have other qualifications. Just because you are a great database engineer, does not mean that you are qualified to make the userinterface. And the man-month your project is assigned does not know the project. Some of the members of the project team will have to spend time training the new guy. In a really bad scenario, Brooks describes a situation where one member of the team have to use a month to train three new team members. The project might have been assigned three man-months. But the first month uses four man-months on training. Placing a net drain of one man-month on the ressources available to the project, compared to the initial situation. After one month, the project is even more delayed. And the project manager now has to explain to his boss why. A difficult question to answer when you have just been assigned more ressources.

Secondly, the tasks performed in a project are not interchangeable. They are usually sequential in nature. First we have to do this, then we have to test it.

Project managers often refers to people who believes that if you assign nine women to the task of carrying a pregnancy to terms – you will become a father next month.

So – what to do? There is no simple answer. The task of management is to pressure the project manager to perform. As a project manager, it is easy to come to the conclusion that they dont understand anything, and that their demands are unreasonable. But maybe, just maybe, you actually could do better.

When you have done better, you are still left with the task of explaining to your boss, that he is not going to be a father next month, just because you get more womanpower. Good luck.

Den her bliver på dansk. For jeg vil skrive lidt om hvordan vi taler om hvor vi er. På dansk. Det er jeg blevet opfordret til af chefen. Så det må jeg hellere gøre.

Der var ingen tvivl. Det hedder oppe på Farma. Og nede på Nørre Alle. Sådan var det i hvert fald da jeg begyndte at arbejde på Farma. Jeg syntes det var meget enkelt. Farma ligger nord for Nørre Alle. Når man kigger på et kort, vender nord opad. Hvis lokation A ligger nord for lokation B, ligger A oppe relativt til B. Og B ligger nede. Det er i fuld overenstemmelse med hvad DRs sprogeksperter mente i 1994. Og hvad man finder på sproget.dk, drevet af Dansk Sprognævn og Det Danske Sprog- og Litteraturselskab. Og dog gav det problemer…

Nogen mente nemlig at Nørre Alle var et vigtigere sted end Farma. Og derfor var det upassende, at jeg talte om “oppe på Farma” og “nede på Nørre Alle”. Det havde vist noget at gøre med hvor de selv sad. Personligt mente jeg at geografien måtte gøre udslaget. Og skulle vi endelig tale om hvilken adresse der var vigtigst – og det kunne vi godt – så gav det samme resultat: “Oppe på Farma”.

Egentlig er det meget enkelt. Reglen for geografiske navne er, at vi siger op(pe) om det der ligger nord for os. Ned(e) om det der ligger mod syd. Og over/ovre, om steder der ligger på samme breddegrad som os selv. Når et sted ligger både nord og østligt for os, så er det i almindelighed oppe der vinder. Som i “oppe på Færøerne”.

Og hvorfor er det så vigtigt? Det er det, fordi op er godt, og ned er dårligt. Åbenbart. Op ad bakke synes jeg nu ikke er så godt. Når prisen på noget går ned er det som regel ikke dårligt, i hvert fald ikke for forbrugerne. Hvis vi bevæger os ind i fantasiens verden, er det selvfølgelig godt hvis ens løn går op, men det kan vi ikke huske hvornår sidst er sket, så det er irrelevant. Med andre ord, om op og ned er godt eller skidt er ikke helt entydigt. Altså bortset fra at det åbenbart betød noget at Farma lå oppe i forhold til Nørre Alle.

Well. Universitetsbiblioteket har mange adresser. Og jeg har siddet på en del af dem. Og hvordan taler vi om dem? Da jeg sad på Diamanten var der ingen tvivl. Alle andre steder var ude. Nogen var måske oppe, andre var vel egentlig nede. Men det ord brugte jeg ikke. Jeg skulle ud på Nørre Alle. Bagefter skulle jeg måske op på Farma – fordi jeg tog fra Nørre Alle til Farma. Men derefter skulle jeg ind på Diamanten igen. Rent geografisk giver det mening. Diamanten ligger i centrum af København. Ikke helt i den gamle middelalderby, men der er ingen tvivl om at der er steder der ligger inden for voldene. Og steder der ligger uden for. Og Diamanten ligger så afgjort indenfor. Sådan da. De andre steder ligger i hvert fald udenfor. Næsten. Så i København, og med et centrum der ligger hvor det nu ligger, var der ikke så meget at rafle om.

Men jeg har nu en fornemmelse af, at Diamanten kunne have ligget en del andre steder. Hvad nu hvis den faktisk havde ligget på Universitetsparken. Og det Farmaceutiske Bibliotek havde ligget inde på Slotsholmen?. Havde det så heddet ude på Diamanten? Jeg tror det, men jeg er ikke sikker. Min usikkerhed skyldes, at Diamanten er magtcentret i biblioteket. Det er der den høje chef sidder. Det er galaksens centrum. Jeg vil ikke helt udelukke at den betydning Den Sorte Diamant har gør at den ville kunne overrule geografien.

Og det er netop det forhold, der gjorde at der blev reageret på at jeg talte om “oppe på Farma”. Det blev forstået som “Farma er vigtigere end det der ligger syd for Farma”. Det er jo objektivt korrekt at det er tilfældet. Men det er altså også en simpel geografisk konstatering. Der var så nogen der havde den fejlagtige opfattelse, at Nørre Alle var så meget vigtigere end Farma, at det kunne overrule geografien.

Der er ingen tvivl om at der ligger en masse magtrelationer i brugen af disse relationelle ord. Jeg tror for eksempel ikke at det er et tilfælde at nord vender opad på kortene. Havde de nordeuropæiske lande ligget syd for ækvator, tror jeg nok vi hurtigt havde fået ændret enten sproget, eller konventionen for hvad der vender op og ned på et landkort.

Men alt det ændrer ikke på at det hedder oppe på Farma. Fordi Farma ligger længst mod nord. Og hvis der er mere brok, vil jeg bare gøre opmærksom på, at Nørre Alle 49 ligger 13,00 meter over havets overflade. Og Universitetsparken 4 ligger 15 centimeter højere.