Other people then have to pull out the data from those spreadsheets.
“Other people” tend to spend a lot of time crying into their coffee.
At the moment, I am trying to pull out data of a spreadsheet, where “something” can have a value of 1, 2 or 3. That is of course marked by an “x” in a cell. I need to convert that x to a number.
That is rather simple. What is not so simple, is that there can be two x’es. One, in black, to denote the current state of affairs. And a second x, in red, to denote what a future, state is wanted to be.
So – I need a way to get the color of an x. VBA can do that:
Function GetColour(ByVal Target As Range) As Single
Application.Volatile
GetColour = Target.Font.Color
End Function
And if I need a logical test:
Function IsBlack(ByVal Target As Range) As Boolean
Application.Volatile
If Target.Font.Color = 0 Then
IsBlack = True
Else
IsBlack = False
End If
End Function
This probably sounds like humble bragging. But I have recently – again – reailized that my biggest weakness is that I take responsibility.
Hey! How is that a weakness?
Well… It becomes a weakness when you continually take responsibility for stuff that is really not your responsibilty. To the extent that you get stress, hypertension and ulcers. And to the extent that it impacts negatively on the things that actually are your responsibility.
And I have just done it again. The ad for a meeting in the local party is not very readable. That is not my responsibility. It belongs to the chairman. Not me. I should simply notify him that it is not very readable. And trust that he will do something about it. Instead I am thinking about remaking it myself. It would not be very difficult. But I do get stressed. If I have to redesign the ad, I wont have time to cook dinner tonight. And clean the house.
This is something that I really have to get better at handling. Otherwise I’ll be a very responsible person, doing great things for people and organizations around me. While burning out very fast.
Crime is a bad thing. No doubt about it. And one of the main topics in todays debate climate is – “those ‘orrible immigrants are very criminal. Look at these numbers, they prove it!”. Usually written with caps-lock engaged.
Well. Maybe they are, and maybe they do. But if you want to use statistics to prove it – pretty please, do not obfuscate the numbers.
This is an example. A blog post from one of the more notable danish newspapers. In the US it would be regarded as communist, in the rest of the world we would think of it as relatively conservative.
The claim is, that the number of reported rapes and other violent crimes in Denmark, are the highest ever. That is because of the increasing numbers of immigrants in Denmark, especially muslims. Use Google translate if you want the details.
Again, that claim might be true. But the graphs in the post, that supposedly documents the claim, are misleading. To say the least.
First – the numbers come from the Danish Statistical Bureau. They have a disclaimer, telling us that changes to the danish penal code, means that a number of sexual offenses have been reclassified as violent crimes since 2013. If the number of violent crimes suddenly includes crimes that did not use to be classified as violent crimes, that number will increase. Not much of a surprise. Yes, the post asks why the numbers are still increasing after that reclassification. One should expect them to level off. And again the post may have a valid point. I don’t know. But what I do know, is that the graphs are misleading.
Heres why. The y-axis has been cut of. Lets recreate the graphs, and take a look.
There are two graphs. The first shows the number of reported cases of rape from 1995 until today.
The second shows the total number of reported cases of violent crimes in the same period. Both sets of data comes from http://www.statistikbanken.dk/.
We’re going to need some libraries:
library(ggplot2)
library(gridExtra)
Lets begin by pulling the data.
There might be better ways, but I’ve simply downloaded the data. Two files:
The last seven lines are the notes about changes in which cases are counted in this statistics. I think that is a pretty important point, but they are difficult to plot.
The graph for rape, as presented in the post, and with a more sensible y-axis:
post <- ggplot(rape, aes(x=V1, y=V2)) +
geom_line(group=1) +
scale_x_discrete(breaks = rape$V1[seq(1, length(rape$V1), by = 20)]) +
theme_classic()
nice <- post + ylim(0,max(rape$V2))
grid.arrange(post, nice, ncol=2)
And the one for violent crimes in general, again with the original on the left, and the better on the right:
post <- ggplot(violence, aes(x=V1, y=V2)) +
geom_line(group=1) +
scale_x_discrete(breaks = violence$V1[seq(1, length(violence$V1), by = 20)]) +
theme_classic()
nice <- post + ylim(0,max(violence$V2))
grid.arrange(post, nice, ncol=2)
So, still, some pretty scary increases. And the change in what is counted should give an increase. But that increase should level off, which it does not. Clearly something is not as it should be. But lets be honest, the graphs on the right are not quite as scary as the ones on the left.
Also – that change in what is counted as sexual assaults – it can explain the initial increase, but then it should level off. That is a fair point. However, there were other things that changed in the period. #metoo for example. I think it would be reasonable to expect that a lot of cases that used to be brushed of as not very important, are now finally being reported. The numbers might actually have leveled off without #metoo.
Anyway, my point is, that if you want to use graphs to support your claims, do NOT cut off the y-axis to make them look more convincing.
First of all, this is in no way a statement on the immigration crisis in Europe. I do have opinions. But it is more a reaction or reflection on three maps I saw on this page.
Danish televison channel TV2 is illustrating the number of refugees or perhaps rather immigrants received in EU-memberstates in the period 2015 to 2017. This is the map showing the number of immigrants to EU in 2015
Note Germany. Germany welcomed the absolutely highest number of immigrants. What piqued my interest though, is that this might be a good illustration of the numbers, it is not really the relevant comparisons. Yes, Germany welcomed more refugees than Denmark did. But Germany is a rather larger country than Denmark. For a given value of “fair”, it is only fair that Germany takes more refugees than smaller countries.
A more relevant comparison might be the number of refugees compared to population. Or area. Sweden saw (at that time) no problems with welcoming a huge number of migrants, because, as they said, there are a lot of un-populated space in Sweden, plenty of room for everyone! Or perhaps GDP is a better way. Richer countries should shoulder a larger part of the challenge than poorer countries.
I’m not concerned here with what is fair. What concerns me is that the graphic is misleading. Lets make an attempt at fixing that. Or at least present a slightly different perspective on the data.
I’ll try to illustrate the number of migrants as a proportion of population in the different countries. The data is “stolen” directly from the news-channel. They have it from UNHCR, Eurostat and the European Parlament.
The first step will be to get the data.
url <- "http://nyheder.tv2.dk/udland/2018-06-28-se-kortet-saa-mange-asylansoegere-har-de-forskellige-eu-lande-taget"
data <- readLines(url)
By inspection, I can see that the relevant data is in these three lines:
There is a small problem. Strange danish characters are encoded. Lets fix that:
library(stringr)
data <- str_replace_all(data,"\\\\u00d8", "Ø")
data <- str_replace_all(data,"\\\\u00e6", "æ")
data <- str_replace_all(data,"\\\\u00f8", "ø")
And the regular expression picking that out of the data ought to be:
‘\“(\p{L}+)\”:{\“valueheat\”:(\d+|\“\”),’
For some reason that is not working. I probably should try to figure that out, but I’m on vacation, and would rather drink cold white wine that dig too deep into the weirdness that is regular expressions in R.
Now, lets get these data into some dataframes. First I’m unlisting the data, then I pour it into a matrix to get the right shape. And then I’m converting the matrices to dataframes:
I’m going to need just one dataframe. I get that by joining the three dataframes:
library(dplyr)
total <- left_join(dat.2015, dat.2016, by="Land")
total <- left_join(total, dat.2017, by="Land")
The numbers are saved as characters. Converting them to numeric:
total$`2015` <- as.numeric(total$`2015`)
## Warning: NAs introduced by coercion
total$`2016` <- as.numeric(total$`2016`)
## Warning: NAs introduced by coercion
total$`2017` <- as.numeric(total$`2017`)
## Warning: NAs introduced by coercion
That introduced some NAs. Countries where there are no data.
Inspecting the data, I can see that there are data for all three years for some countries. For other countries, there are no data at all. The function complete.cases() will return true for a row without NAs.
Using that to get rid of countries where we don’t have complete data:
And while I’m at it, the second line gets rid of the factors, and the third removes the thousand separators (“.”)
Now I can join the dataframe containing population figures, with the dataframe containing countries and number of migrants:
total <- left_join(total, tabellen, by="Land")
## Warning: Column `Land` joining character vector and factor, coercing into
## character vector
There are three smaller problems. Cyprys, France and Ireland. The problem is that the country name I get from Wikipedia contains a note. I might be able to get rid of that by code. I’m going to do it manually.
Now I have a nice dataframe with the name of the countries (in danish), the numbe of migrants received in 2015, 2016 and 2017, and the population in 2018.
Now it is time to look at some maps.
library(ggplot2)
library(rworldmap)
## ### Welcome to rworldmap ###
## For a short introduction type : vignette('rworldmap')
I am going to match the countries in my dataframe, with the countries I get from the map data. That requires that I have the english names for the countries in my dataframe.
That retrieves data for the entire world. I’m only interested in EU:
EU <- worldmap[which(worldmap$NAME %in% enland),]
EU <- map_data(EU)
##
## Attaching package: 'maps'
## The following object is masked from 'package:purrr':
##
## map
The first line extracts the part of the world map that has names in the list of countries that I have data for.
map_data() converts that into a nice structure that is suitable for entering into ggplot.
Next step is calculating the number of migrants received in each country as a proportion of that countrys population:
total <- total %>%
mutate(`2015` = `2015`/Population*100, `2016` = `2016`/Population*100, `2017`=`2017`/Population*100)
I’m mutating the columns 2015-2017 by dividing by population. And multiplying by 100 to get percentages.
The almost final step, is to join my migrant-proportions with the map data:
total <- left_join(total,EU, by=c("enland"="region") )
The map data does not call the countries for countries. Rather their names are saved in the variable “region”.
And now the final step. I’m going to need the data on tidy form. So I’m loading tidyr.
Then I pass the data frame to select(), where I pick out the variables I need. Long(itude), lat(itude), 2015, 2016 and 2017, and the name of the country.
That is passed to gather(), where I make a new row for each year, with the proportions in the new variabels year and prop.
All that is passed to ggplot, and a layer where the polygons describing the countries are plotted. They are with a colour matching the proportions. And grouped by “group”. This is important. Grouping by country name gives weird results. I’ll get back to that. color=“white” plots the lines in the polygons in white.
Finally, I facet the data on year.
library(tidyr)
total %>%
select(long,lat,`2015`,`2016`,`2017`, group) %>%
gather(year, prop, `2015`:`2017`)%>%
ggplot() +
geom_polygon(aes(long, lat, fill = prop, group = group), color = "white") +
theme_void() +
facet_wrap(~year, ncol=2)
Thats it!
And now the picture is slightly different. What is interesting is that Germany still takes a higher proportion of the migrants than other countries. But in 2015, they didn’t. That was the year when the german chancellor Angela Merkel said the famous words “Wir schaffen das”, We’ll manage. But also the year when Hungary and sweden welcomed migrants in numbers equalling 1.79% and 1.65% of their population respectively. You can compare that with the fact that Germany the same year received migrants equalling 0.58% of their population.
A cynic might claim that it is no surprise that Sweden and Hungary closed their borders late in 2015.
Any way, that is a different subject. I just think that these three maps are slightly more informative than what TV2 provided.
Also, I promised to get back to the group thingy.
Making the same plot, but grouping on country names:
total %>%
select(long,lat,`2015`,`2016`,`2017`, group, enland) %>%
gather(year, prop, `2015`:`2017`)%>%
ggplot() +
geom_polygon(aes(long, lat, fill = prop, group = enland), color = "white") +
theme_void()
What happens is that the polygons describing Italy are grouped in a way that connects the parts describing sicily to the northern part of Italy. That looks weird. The same happens with Sardinia.
Finally. I have not been very consistent in my use of words. I have used “received” and “welcomed” interchangeably. Hungary and Denmark has not been very welcoming. But we are talking about real humans here, and welcoming simply sounds nicer than received. Complicating the situation was the fact that a lot of the arrivals were not actually what we would normally call refugees. At least not refugees from war. So I have also not been consistent in the use of “migrant” vs “refugee”. That is not really my point. The point is that we should always think about how these kinds of numbers are presented.
Squarefree Binomial Coefficients. Building Pascals triangle to 51 rows (look it up – its a bloody nightmare to write out here): Locate all distinct numbers in that triangle, that does not divide by a square of a prime.
OK. First step is to get all the numbers in 51 rows of Pascals triangle.
R has a built in function, choose(i,j) that returns the number for row i, position j.
We can use that to iterate through all the possible positions in a 51 row triangle:
numbers <- 1
for(i in 0:50){
for(j in 0:i){
numbers <- c(numbers, choose(i,j))
}
}
Next step is to make sure that we only have unique, distinct, values:
numbers <- unique(numbers)
We now need to divide each number by the square of a prime. My first instinct was to generate all primes smaller than the squareroot of the largest number in numbers.
That would be all primes lower than 11,243,247.
I would then square all those primes, and see if one of them divided neatly into the numbers in the triangel.
Thats an awfull lot of primes.
Much easier would be to note, that if the square of a prime divides neatly into a number, then the prime does as well. And is in fact a prime factor in that number.
And since we have a nice library that makes it easy to get the primefactors, thats the way to do it.
Passing the numbers vector to the discard function. Discard the element, if the element, modulo the primefactors in that element squared, has one (or more) results that are equal to 0.
The answer is the sum of the elements that are left. Plus 1, since 1 is discarded by the function. factors(1) – for some reason – returns 1.
Project Euler, problem 263 – An engineers’ dream come true
This is, at the time of coding, the highest numbered Project Euler problem I’ve tried to tackle. With a difficulty rating of 75% it is also the most difficult. At least on paper. But An engineers’ dream come true? How can I not, as an engineer, not try to solve it?
We har looking for an n, for which it holds that:
n-9 and n-3 must be consecutive primes
n-3 and n+3 must also be consecutive primes
n+3 and n+9 must also be consecutive primes.
These are primepairs that are “sexy”, that is that have differences of 6.
Also, n, n-8, n-4, n+4 and n+8 must be practical numbers, that is numbers where the numbes 1:n can be written as sums of distinct divisors of n.
So if a number n gives sexy prime pairs, and are very practical – that is what an engineer dreams of – hence the paradise.
The fastest way seems to be to generate a list of primes, sieve those out that conforms to the requirements for consecutive primes, and then test those numbers for practicality.
Lets get started!
The trusty numbers library provides the primes, up to 1000000. Then for each of those primes, return the n-value, if the differences between the sets of primes, are 6.
What we get from this, is not primenumbers, but values of n, that gives the consecutive primes we need.
Now I have a list of candidate n’s based on the requirements for primes. Next I need to check for practicality.
First I tried a naive way. Generate the list of numbers that I need to sum to using distinct divisors, for a given x.
Then get the divisors of x. I dont need to check if I can sum to the numbers that are themselves divisors, so excluding them leaves me with at slightly smaller set. Then I get all the ways I can take two divisors from the set of divisors. Sum them, and exclude them from the list of numbers. I continue until I have taken all the possible combinations of 2, 3, 4 etc divisors, based on how many there are. If there are no numbers left in the vector of numbers that I need to be able to sum to, I was able to express all those numbers as sums of distinct divisors. And then x was a practical number.
practical <- function(x){
test <- 1:x
divs <- divisors(x)
test <- setdiff(test,divs)
for(i in 2:length(divs)){
test <- setdiff(test,combn(divs,i,sum))
}
!as.logical(length(test))
}
Two problems. One that can be fixed. I continue to generate combinations of divisors and summing them, even if I have already found ways to sum all the numbers. The second problem is more serious. When I have to test a number with really many divisors – it takes a long time. Also, handling a vector containing all numbers 1:1000000 takes a lot of time.
I need a faster way of checking for practicality.
Wikipedia to the rescue. There exists a way of checking. I have no idea why it works. But it does.
For a number x, larger than 1, and where the first primefactor is 2. All primefactors are ordered. Taking each primefactor, that has to be smaller than or equal to the sum of the divisors of the product of all the smaller primefactors. Plus one. Oh! And that sum – if 3 is a primefactor twice, that is if 32 is a factor, I should square 3 in the product.
That sounds simple.
For a number x, get the primefactors. Use table to get the counts of the primefactors, ie that 3 is there twice. Those are the values of the result from the table function. The names of the table function are the primefactors.
For each factor from number 2 to the end of the number of factors, get the names of the primefactors from number 1 to just before that factor we are looking at (as numeric). Exponentiate with the values from the table – that is how many times a primefactor is a primefactor. Generate the product, get the divisors of that product, sum them, and add 1. If the factor we were looking at is larger that that, we have determined that x is not practical – and can return FALSE. If x gets through that process, it is practial.
I need to handle the case where there is only one primefactor – 2. Those numbers are practial, but the way I have done the check breaks when there is only one primefactor. Its simple enough, just check if there is only one distinct primefactor, and return TRUE in that case.
Now I can take my candidate n’s based on the requirements for primepairs, and just keep the n’s that are themselves practical. And where n-8, n-4, n+4 and n+8 are also practial:
This is kinda problem. The n we are looking for are actually pretty large. I know this, because this writeup is written after I found the solution. So it is not because the code is flawed.
Nu har vi så den udfordring, at vi skal have fat i ret høje tal.
Now – I could just test larger and larger sets of primes. I run into memory problems when I try that. Then I could write some code, that generates primenumbers between some two numbers, and increse those numbers until I get number four.
I have not. Instead I just tried again and again, until I found number 4. We need to have an upper limit for primes that are some four times larger than the one I just used. Anyway, I found number four. And got the green tickmark.
Lesson learned
Sometimes you really need to look into the math. This would have been impossible if I had not found a quicker way to check for practicality.
Also, I should refactor this code. The check for practicality could certainly be optimised a bit.
And – I should probably check for primality, rather than generating the list of primes.
Including their catalogue. I was wondering… Would it be possible to build a machine learning algorithm, that returns a subject, based on title? And maybe other information?
Something to look into during the long hours in the summer, where the boss is on holiday, the patrons are away, and we have time to do interesting stuff? Not that we are not doing interesting stuff already, but stuff that is interesting in it self.
In a world, changing with increasing speed, it is simply human nature to try to make things stay the way they have always been. We don’t like change. And we are prepared to do a lot of work to prevent the change. Even if it is inevitable.
Because of this, forward thinking managers, not only in libraries, are spending a lot of time changing things. That is usually a good thing.
But sometimes it is not. The idea that we should change, adopt, develop and reform things, because if we don’t the world around us will change, can often lead to spending a lot of time running forward in circles. And not necessarily forward. We are not always certain that a change is actually for the better, but at least it is a change, and we have to change don’t we?
So – one of the cynical lessons from someone who has spent the last 20+ years observing projects, is that far too often, changes are made based on the assumption that change in itself is good. Or:
To improve things, things must change
We are changing things
Therefore, we are improving things.
Before you spend my tax money on changing things, please pause and consider if the change is actually for the better. Or just a change.
We spend a lot of time talking about it. And to be honest, in my opinion, not very much time actually doing anything about it.
Competence is the ability to do something successfully or efficiently.
It follows logically, that when you develop your competences, you will, afterwards, be able to do something (successfully) that you were not able to do before. Or you will be able to do it more efficiently, ie better or faster.
Usually it is not enough to acquire a skill. “Use it or lose it” they say. I was actually speaking pretty good german when I was 15. Today I’m anything but fluent, and struggling with Duolingo to regain the skills I had 28 years ago. I do read technical german pretty well. And that testifies to the fact, that I have used the part of my “german competences” that relates to technical texts. Whereas I have not used the part that allows me to speak or understand spoken, german.
It’s the same with the skills you need at your job. You may acquire some skills at a course, by reading a book, by getting peer-training. But if you don’t use them afterwards, you loose them.
Far too often we spend time attending a course, get back to work, and not use the things we learned. The net result is that we wasted the time and money spend on the course. Noone made sure that we had the time to use our new skills. Or that procedures in the organization was changed accordingly. Maybe we learned skills that we don’t really need.
It would be fun to learn to weld. And even better if my boss would pay for it. But I would never get to use that skill at work. Or at home. So it would be a bloody waste of time! It would probably be bloody in a literal sense as well…
That is one problem. Developing skills and competences that are not actually useful. Or used. I have had it happen to me several times. I’ve even paid for useless courses and activities out of my own pocket.
The other problem is labelling.
We all agree that competence development is positive. We want more of it. It is nice, and necessary. Therefore it is not surprising, that we would like to label activities that does not really develop any competences, as developing competences. I have been required to register, as competence development, giving an introduction to ebook readers. I agree that the participants in the course, the colleagues subjected to me talking about ebook readers for 1½ hours, had their competences developed. Or at least I hope they had.
But how is giving that introduction developing my competences? As an academic I am able to argue for, or againts, anything, sometimes even at the same time. But my competences regarding ebook readers were definitely not developed. I might have gotten slightly better at giving introductions. But I did not learn anything new about ebook readers.
Why do I care? Well. First of all I abhor newspeak. If something is not developing competences, dont claim that that is what is happening. It might be interesting. It might be beneficial. But if competences are not developed, they are not developed, and please do not claim that they were.
Secondly, it generates the impression that we are spending a lot of resources on developing competences. That is a problem if that impression is false. We will wake up some day, and wonder why we did not learn what was necessary to survive in a changing library landscape. It will be a mystery to us, because we thought we were spending a lot of time developing all sorts of competences. But in reality we did not develop anything.
Anyway. Assume we have five cases, with two values for a given treatment, each labelled “AVO” and “INT”. How to get a barchart, where each cases is one bar, and the bars are coloured based on the percentages of the two values?
## cases treatment value
## 1 1 AVO 0.9769620
## 2 2 AVO 0.3739160
## 3 3 AVO 0.7615020
## 4 4 AVO 0.8224916
## 5 5 AVO 0.5735444
## 6 1 INT 0.6914124
## 7 2 INT 0.3890619
## 8 3 INT 0.4689460
## 9 4 INT 0.5433097
## 10 5 INT 0.9248920
OK, nice and tidy data.
What I want is a bar for case 1, filled with percentages. One color with 0.976/(0.976+0.691) and another color for 0.691/(0.976+0.691).
library(ggplot2)
ggplot(df)+
geom_col(aes(y=value, x=cases, fill=treatment), position = "fill")
We’re using geom_col. That is a shorthand for geom_bar with stat=“identity”, and is the shortcut to expressing values in the barchart.
The y-value is the value from df, the x-value is the cases. And we’re colouring the bars based on the value in treatment. The position=“fill” stack the bars, and normalize the heights. If we wanted to just stack the elements, we would use position=“stack”:
ggplot(df)+
geom_col(aes(y=value, x=cases, fill=treatment), position = "stack")
This is a great, like really great, cheatsheet for ggplot:
