Hvis man skal arbejde sammen med mig. Et godt spørgsmål jeg fik forleden.
Man skal holde hvad man lover. Og hvis man ikke kan, så skal man sådan set bare sige til, for jeg er utroligt tilgivende. Men hvis du lover noget, og ikke fortæller at det desværre ikke kan lade sig gøre, så bliver jeg træt når det står klart at du ikke leverer.
Hvad ellers? Der hvor jeg virkelig bliver træt af folk er når de er inkonsistente. Eller hykleriske om man vil.
Du må godt være religionskritisk. Du må også godt være islamkritisk. Men hvis du påstår at du er religionskritisk, så bliver du dæleme nødt til faktisk at være det. Hvis du hævder at være religionskritisk, men pudsigt nok kun er kritisk overfor islam. Så bliver jeg lidt træt af dig.
Ret præcist lige så træt som jeg bliver hvis du hævder at være religionskritisk. Men tilfældigvis ikke overfor islam.
Hvis du synes det er urimeligt at man sætter etiketter på folk uden at have fået lov til det af dem. Så lad være med at sætte etiketter på mig uden at spørge. Du bryder dig ikke om at blive kaldt transseksuel. Det hedder transkønnet. Fint med mig, ingen problemer. Men hvis du bruger vigtigheden af ikke at sætte uønskede etiketter på folk som argument for det. Så lad være med at kalde mig cis-kønnet (uden at spørge om lov først).
Du må godt indføre burkaforbud. Men lad være med at påstå at du er liberal når du gør det.
Du må godt forbyde sombreroer til fester på Københavns Universitet fordi nogen bliver krænkede. Husk blot også at forbyde t-shirts med billeder af Che Guevara – du ved, ham der slog et signifikant tre-cifret antal mennesker ihjel under udbredelsen af en totalitær ideologi. Og satte homosexuelle i koncentrationslejre.
Misforstå mig ret. Jeg bliver også træt af folk der er islamkritiske. Jeg bliver bare mere træt af dem, hvis de – i modstrid med alt hvad de faktisk gør – hævder at de skam er religionskritiske.
Jeg synes jo i den grad at man skal være utroligt forsigtig med at sætte etiketter på folk. Men hvis du også synes det – så lad være med selv at gå rundt og etikettere folk.
Og jeg er heller ikke fan af burkaforbud (eller for den sags skyld burkaer). Men der burde være en paragraf i markedsføringsloven der ramte Danmarks “Liberale” Parti, når de indfører det.
Og du må for min skyld godt indføre forbud mod krænkende sombreroer. Men vær dog ærlig om at det handler om at du forbyder ting som et bestemt politisk segment ikke bryder sig om. For lur mig om netop Che Guevara t-shirts ikke vil blive ramt af forbud, skulle nogen føle sig krænket af dem.
A rather popular chart type. Not really my favorite, but I can see how it makes things easier to understand for people who are not used to read and understand charts. The reason for my less than favourable view on waffle charts are probably linked to its overuse in meaningless infographics.
A waffle chart is a grid with squares/cells/icons/whatever, where each cell represents a number of something.
One annoyance: waffle wants you to spell colours wrong.
waffle takes a named vector of values, rows sets the number of rows of blocks. Default is 10.
One standard way, is to show a 10×10 grid, where each cell represents 1% of the total:
waffle(vec/sum(vec)*100)
Bloody annoying – waffle rounds the values of the vector, leading to only 98 squares. So you have to manipulate your vector to get to 100. Well, actually it is probably a minor annoyance.
What if you want something else than coloured squares?
The arguments “use_glyph” and “glyph_size” makes that possible.
First, we’ll need the library extrafont
library(extrafont)
We’ll also need to have the “awesomefonts” installed. It can be downloaded from:
This should be easier if you are on a desktop machine. As I’m running this through my own installation of RStudio on a remote server, it was a bit more difficult.
I needed to place the “fontawesome.ttf” file in the “/usr/share/fonts/truetype/fontawesome” directory.
Then, running R as superuser on the commandline, I imported the extrafont library, and then ran “font_import()”.
But then it worked!
There is now a long list of 593 different icons, that can be used. If you want a list, just run fa_list().
And now, we can make a waffle chart with the glyph of our choice.
Other people then have to pull out the data from those spreadsheets.
“Other people” tend to spend a lot of time crying into their coffee.
At the moment, I am trying to pull out data of a spreadsheet, where “something” can have a value of 1, 2 or 3. That is of course marked by an “x” in a cell. I need to convert that x to a number.
That is rather simple. What is not so simple, is that there can be two x’es. One, in black, to denote the current state of affairs. And a second x, in red, to denote what a future, state is wanted to be.
So – I need a way to get the color of an x. VBA can do that:
Function GetColour(ByVal Target As Range) As Single
Application.Volatile
GetColour = Target.Font.Color
End Function
And if I need a logical test:
Function IsBlack(ByVal Target As Range) As Boolean
Application.Volatile
If Target.Font.Color = 0 Then
IsBlack = True
Else
IsBlack = False
End If
End Function
This probably sounds like humble bragging. But I have recently – again – reailized that my biggest weakness is that I take responsibility.
Hey! How is that a weakness?
Well… It becomes a weakness when you continually take responsibility for stuff that is really not your responsibilty. To the extent that you get stress, hypertension and ulcers. And to the extent that it impacts negatively on the things that actually are your responsibility.
And I have just done it again. The ad for a meeting in the local party is not very readable. That is not my responsibility. It belongs to the chairman. Not me. I should simply notify him that it is not very readable. And trust that he will do something about it. Instead I am thinking about remaking it myself. It would not be very difficult. But I do get stressed. If I have to redesign the ad, I wont have time to cook dinner tonight. And clean the house.
This is something that I really have to get better at handling. Otherwise I’ll be a very responsible person, doing great things for people and organizations around me. While burning out very fast.
Crime is a bad thing. No doubt about it. And one of the main topics in todays debate climate is – “those ‘orrible immigrants are very criminal. Look at these numbers, they prove it!”. Usually written with caps-lock engaged.
Well. Maybe they are, and maybe they do. But if you want to use statistics to prove it – pretty please, do not obfuscate the numbers.
This is an example. A blog post from one of the more notable danish newspapers. In the US it would be regarded as communist, in the rest of the world we would think of it as relatively conservative.
The claim is, that the number of reported rapes and other violent crimes in Denmark, are the highest ever. That is because of the increasing numbers of immigrants in Denmark, especially muslims. Use Google translate if you want the details.
Again, that claim might be true. But the graphs in the post, that supposedly documents the claim, are misleading. To say the least.
First – the numbers come from the Danish Statistical Bureau. They have a disclaimer, telling us that changes to the danish penal code, means that a number of sexual offenses have been reclassified as violent crimes since 2013. If the number of violent crimes suddenly includes crimes that did not use to be classified as violent crimes, that number will increase. Not much of a surprise. Yes, the post asks why the numbers are still increasing after that reclassification. One should expect them to level off. And again the post may have a valid point. I don’t know. But what I do know, is that the graphs are misleading.
Heres why. The y-axis has been cut of. Lets recreate the graphs, and take a look.
There are two graphs. The first shows the number of reported cases of rape from 1995 until today.
The second shows the total number of reported cases of violent crimes in the same period. Both sets of data comes from http://www.statistikbanken.dk/.
We’re going to need some libraries:
library(ggplot2)
library(gridExtra)
Lets begin by pulling the data.
There might be better ways, but I’ve simply downloaded the data. Two files:
The last seven lines are the notes about changes in which cases are counted in this statistics. I think that is a pretty important point, but they are difficult to plot.
The graph for rape, as presented in the post, and with a more sensible y-axis:
post <- ggplot(rape, aes(x=V1, y=V2)) +
geom_line(group=1) +
scale_x_discrete(breaks = rape$V1[seq(1, length(rape$V1), by = 20)]) +
theme_classic()
nice <- post + ylim(0,max(rape$V2))
grid.arrange(post, nice, ncol=2)
And the one for violent crimes in general, again with the original on the left, and the better on the right:
post <- ggplot(violence, aes(x=V1, y=V2)) +
geom_line(group=1) +
scale_x_discrete(breaks = violence$V1[seq(1, length(violence$V1), by = 20)]) +
theme_classic()
nice <- post + ylim(0,max(violence$V2))
grid.arrange(post, nice, ncol=2)
So, still, some pretty scary increases. And the change in what is counted should give an increase. But that increase should level off, which it does not. Clearly something is not as it should be. But lets be honest, the graphs on the right are not quite as scary as the ones on the left.
Also – that change in what is counted as sexual assaults – it can explain the initial increase, but then it should level off. That is a fair point. However, there were other things that changed in the period. #metoo for example. I think it would be reasonable to expect that a lot of cases that used to be brushed of as not very important, are now finally being reported. The numbers might actually have leveled off without #metoo.
Anyway, my point is, that if you want to use graphs to support your claims, do NOT cut off the y-axis to make them look more convincing.
First of all, this is in no way a statement on the immigration crisis in Europe. I do have opinions. But it is more a reaction or reflection on three maps I saw on this page.
Danish televison channel TV2 is illustrating the number of refugees or perhaps rather immigrants received in EU-memberstates in the period 2015 to 2017. This is the map showing the number of immigrants to EU in 2015
Note Germany. Germany welcomed the absolutely highest number of immigrants. What piqued my interest though, is that this might be a good illustration of the numbers, it is not really the relevant comparisons. Yes, Germany welcomed more refugees than Denmark did. But Germany is a rather larger country than Denmark. For a given value of “fair”, it is only fair that Germany takes more refugees than smaller countries.
A more relevant comparison might be the number of refugees compared to population. Or area. Sweden saw (at that time) no problems with welcoming a huge number of migrants, because, as they said, there are a lot of un-populated space in Sweden, plenty of room for everyone! Or perhaps GDP is a better way. Richer countries should shoulder a larger part of the challenge than poorer countries.
I’m not concerned here with what is fair. What concerns me is that the graphic is misleading. Lets make an attempt at fixing that. Or at least present a slightly different perspective on the data.
I’ll try to illustrate the number of migrants as a proportion of population in the different countries. The data is “stolen” directly from the news-channel. They have it from UNHCR, Eurostat and the European Parlament.
The first step will be to get the data.
url <- "http://nyheder.tv2.dk/udland/2018-06-28-se-kortet-saa-mange-asylansoegere-har-de-forskellige-eu-lande-taget"
data <- readLines(url)
By inspection, I can see that the relevant data is in these three lines:
There is a small problem. Strange danish characters are encoded. Lets fix that:
library(stringr)
data <- str_replace_all(data,"\\\\u00d8", "Ø")
data <- str_replace_all(data,"\\\\u00e6", "æ")
data <- str_replace_all(data,"\\\\u00f8", "ø")
And the regular expression picking that out of the data ought to be:
‘\“(\p{L}+)\”:{\“valueheat\”:(\d+|\“\”),’
For some reason that is not working. I probably should try to figure that out, but I’m on vacation, and would rather drink cold white wine that dig too deep into the weirdness that is regular expressions in R.
Now, lets get these data into some dataframes. First I’m unlisting the data, then I pour it into a matrix to get the right shape. And then I’m converting the matrices to dataframes:
I’m going to need just one dataframe. I get that by joining the three dataframes:
library(dplyr)
total <- left_join(dat.2015, dat.2016, by="Land")
total <- left_join(total, dat.2017, by="Land")
The numbers are saved as characters. Converting them to numeric:
total$`2015` <- as.numeric(total$`2015`)
## Warning: NAs introduced by coercion
total$`2016` <- as.numeric(total$`2016`)
## Warning: NAs introduced by coercion
total$`2017` <- as.numeric(total$`2017`)
## Warning: NAs introduced by coercion
That introduced some NAs. Countries where there are no data.
Inspecting the data, I can see that there are data for all three years for some countries. For other countries, there are no data at all. The function complete.cases() will return true for a row without NAs.
Using that to get rid of countries where we don’t have complete data:
And while I’m at it, the second line gets rid of the factors, and the third removes the thousand separators (“.”)
Now I can join the dataframe containing population figures, with the dataframe containing countries and number of migrants:
total <- left_join(total, tabellen, by="Land")
## Warning: Column `Land` joining character vector and factor, coercing into
## character vector
There are three smaller problems. Cyprys, France and Ireland. The problem is that the country name I get from Wikipedia contains a note. I might be able to get rid of that by code. I’m going to do it manually.
Now I have a nice dataframe with the name of the countries (in danish), the numbe of migrants received in 2015, 2016 and 2017, and the population in 2018.
Now it is time to look at some maps.
library(ggplot2)
library(rworldmap)
## ### Welcome to rworldmap ###
## For a short introduction type : vignette('rworldmap')
I am going to match the countries in my dataframe, with the countries I get from the map data. That requires that I have the english names for the countries in my dataframe.
That retrieves data for the entire world. I’m only interested in EU:
EU <- worldmap[which(worldmap$NAME %in% enland),]
EU <- map_data(EU)
##
## Attaching package: 'maps'
## The following object is masked from 'package:purrr':
##
## map
The first line extracts the part of the world map that has names in the list of countries that I have data for.
map_data() converts that into a nice structure that is suitable for entering into ggplot.
Next step is calculating the number of migrants received in each country as a proportion of that countrys population:
total <- total %>%
mutate(`2015` = `2015`/Population*100, `2016` = `2016`/Population*100, `2017`=`2017`/Population*100)
I’m mutating the columns 2015-2017 by dividing by population. And multiplying by 100 to get percentages.
The almost final step, is to join my migrant-proportions with the map data:
total <- left_join(total,EU, by=c("enland"="region") )
The map data does not call the countries for countries. Rather their names are saved in the variable “region”.
And now the final step. I’m going to need the data on tidy form. So I’m loading tidyr.
Then I pass the data frame to select(), where I pick out the variables I need. Long(itude), lat(itude), 2015, 2016 and 2017, and the name of the country.
That is passed to gather(), where I make a new row for each year, with the proportions in the new variabels year and prop.
All that is passed to ggplot, and a layer where the polygons describing the countries are plotted. They are with a colour matching the proportions. And grouped by “group”. This is important. Grouping by country name gives weird results. I’ll get back to that. color=“white” plots the lines in the polygons in white.
Finally, I facet the data on year.
library(tidyr)
total %>%
select(long,lat,`2015`,`2016`,`2017`, group) %>%
gather(year, prop, `2015`:`2017`)%>%
ggplot() +
geom_polygon(aes(long, lat, fill = prop, group = group), color = "white") +
theme_void() +
facet_wrap(~year, ncol=2)
Thats it!
And now the picture is slightly different. What is interesting is that Germany still takes a higher proportion of the migrants than other countries. But in 2015, they didn’t. That was the year when the german chancellor Angela Merkel said the famous words “Wir schaffen das”, We’ll manage. But also the year when Hungary and sweden welcomed migrants in numbers equalling 1.79% and 1.65% of their population respectively. You can compare that with the fact that Germany the same year received migrants equalling 0.58% of their population.
A cynic might claim that it is no surprise that Sweden and Hungary closed their borders late in 2015.
Any way, that is a different subject. I just think that these three maps are slightly more informative than what TV2 provided.
Also, I promised to get back to the group thingy.
Making the same plot, but grouping on country names:
total %>%
select(long,lat,`2015`,`2016`,`2017`, group, enland) %>%
gather(year, prop, `2015`:`2017`)%>%
ggplot() +
geom_polygon(aes(long, lat, fill = prop, group = enland), color = "white") +
theme_void()
What happens is that the polygons describing Italy are grouped in a way that connects the parts describing sicily to the northern part of Italy. That looks weird. The same happens with Sardinia.
Finally. I have not been very consistent in my use of words. I have used “received” and “welcomed” interchangeably. Hungary and Denmark has not been very welcoming. But we are talking about real humans here, and welcoming simply sounds nicer than received. Complicating the situation was the fact that a lot of the arrivals were not actually what we would normally call refugees. At least not refugees from war. So I have also not been consistent in the use of “migrant” vs “refugee”. That is not really my point. The point is that we should always think about how these kinds of numbers are presented.
Squarefree Binomial Coefficients. Building Pascals triangle to 51 rows (look it up – its a bloody nightmare to write out here): Locate all distinct numbers in that triangle, that does not divide by a square of a prime.
OK. First step is to get all the numbers in 51 rows of Pascals triangle.
R has a built in function, choose(i,j) that returns the number for row i, position j.
We can use that to iterate through all the possible positions in a 51 row triangle:
numbers <- 1
for(i in 0:50){
for(j in 0:i){
numbers <- c(numbers, choose(i,j))
}
}
Next step is to make sure that we only have unique, distinct, values:
numbers <- unique(numbers)
We now need to divide each number by the square of a prime. My first instinct was to generate all primes smaller than the squareroot of the largest number in numbers.
That would be all primes lower than 11,243,247.
I would then square all those primes, and see if one of them divided neatly into the numbers in the triangel.
Thats an awfull lot of primes.
Much easier would be to note, that if the square of a prime divides neatly into a number, then the prime does as well. And is in fact a prime factor in that number.
And since we have a nice library that makes it easy to get the primefactors, thats the way to do it.
Passing the numbers vector to the discard function. Discard the element, if the element, modulo the primefactors in that element squared, has one (or more) results that are equal to 0.
The answer is the sum of the elements that are left. Plus 1, since 1 is discarded by the function. factors(1) – for some reason – returns 1.
Project Euler, problem 263 – An engineers’ dream come true
This is, at the time of coding, the highest numbered Project Euler problem I’ve tried to tackle. With a difficulty rating of 75% it is also the most difficult. At least on paper. But An engineers’ dream come true? How can I not, as an engineer, not try to solve it?
We har looking for an n, for which it holds that:
n-9 and n-3 must be consecutive primes
n-3 and n+3 must also be consecutive primes
n+3 and n+9 must also be consecutive primes.
These are primepairs that are “sexy”, that is that have differences of 6.
Also, n, n-8, n-4, n+4 and n+8 must be practical numbers, that is numbers where the numbes 1:n can be written as sums of distinct divisors of n.
So if a number n gives sexy prime pairs, and are very practical – that is what an engineer dreams of – hence the paradise.
The fastest way seems to be to generate a list of primes, sieve those out that conforms to the requirements for consecutive primes, and then test those numbers for practicality.
Lets get started!
The trusty numbers library provides the primes, up to 1000000. Then for each of those primes, return the n-value, if the differences between the sets of primes, are 6.
What we get from this, is not primenumbers, but values of n, that gives the consecutive primes we need.
Now I have a list of candidate n’s based on the requirements for primes. Next I need to check for practicality.
First I tried a naive way. Generate the list of numbers that I need to sum to using distinct divisors, for a given x.
Then get the divisors of x. I dont need to check if I can sum to the numbers that are themselves divisors, so excluding them leaves me with at slightly smaller set. Then I get all the ways I can take two divisors from the set of divisors. Sum them, and exclude them from the list of numbers. I continue until I have taken all the possible combinations of 2, 3, 4 etc divisors, based on how many there are. If there are no numbers left in the vector of numbers that I need to be able to sum to, I was able to express all those numbers as sums of distinct divisors. And then x was a practical number.
practical <- function(x){
test <- 1:x
divs <- divisors(x)
test <- setdiff(test,divs)
for(i in 2:length(divs)){
test <- setdiff(test,combn(divs,i,sum))
}
!as.logical(length(test))
}
Two problems. One that can be fixed. I continue to generate combinations of divisors and summing them, even if I have already found ways to sum all the numbers. The second problem is more serious. When I have to test a number with really many divisors – it takes a long time. Also, handling a vector containing all numbers 1:1000000 takes a lot of time.
I need a faster way of checking for practicality.
Wikipedia to the rescue. There exists a way of checking. I have no idea why it works. But it does.
For a number x, larger than 1, and where the first primefactor is 2. All primefactors are ordered. Taking each primefactor, that has to be smaller than or equal to the sum of the divisors of the product of all the smaller primefactors. Plus one. Oh! And that sum – if 3 is a primefactor twice, that is if 32 is a factor, I should square 3 in the product.
That sounds simple.
For a number x, get the primefactors. Use table to get the counts of the primefactors, ie that 3 is there twice. Those are the values of the result from the table function. The names of the table function are the primefactors.
For each factor from number 2 to the end of the number of factors, get the names of the primefactors from number 1 to just before that factor we are looking at (as numeric). Exponentiate with the values from the table – that is how many times a primefactor is a primefactor. Generate the product, get the divisors of that product, sum them, and add 1. If the factor we were looking at is larger that that, we have determined that x is not practical – and can return FALSE. If x gets through that process, it is practial.
I need to handle the case where there is only one primefactor – 2. Those numbers are practial, but the way I have done the check breaks when there is only one primefactor. Its simple enough, just check if there is only one distinct primefactor, and return TRUE in that case.
Now I can take my candidate n’s based on the requirements for primepairs, and just keep the n’s that are themselves practical. And where n-8, n-4, n+4 and n+8 are also practial:
This is kinda problem. The n we are looking for are actually pretty large. I know this, because this writeup is written after I found the solution. So it is not because the code is flawed.
Nu har vi så den udfordring, at vi skal have fat i ret høje tal.
Now – I could just test larger and larger sets of primes. I run into memory problems when I try that. Then I could write some code, that generates primenumbers between some two numbers, and increse those numbers until I get number four.
I have not. Instead I just tried again and again, until I found number 4. We need to have an upper limit for primes that are some four times larger than the one I just used. Anyway, I found number four. And got the green tickmark.
Lesson learned
Sometimes you really need to look into the math. This would have been impossible if I had not found a quicker way to check for practicality.
Also, I should refactor this code. The check for practicality could certainly be optimised a bit.
And – I should probably check for primality, rather than generating the list of primes.
Including their catalogue. I was wondering… Would it be possible to build a machine learning algorithm, that returns a subject, based on title? And maybe other information?
Something to look into during the long hours in the summer, where the boss is on holiday, the patrons are away, and we have time to do interesting stuff? Not that we are not doing interesting stuff already, but stuff that is interesting in it self.
In a world, changing with increasing speed, it is simply human nature to try to make things stay the way they have always been. We don’t like change. And we are prepared to do a lot of work to prevent the change. Even if it is inevitable.
Because of this, forward thinking managers, not only in libraries, are spending a lot of time changing things. That is usually a good thing.
But sometimes it is not. The idea that we should change, adopt, develop and reform things, because if we don’t the world around us will change, can often lead to spending a lot of time running forward in circles. And not necessarily forward. We are not always certain that a change is actually for the better, but at least it is a change, and we have to change don’t we?
So – one of the cynical lessons from someone who has spent the last 20+ years observing projects, is that far too often, changes are made based on the assumption that change in itself is good. Or:
To improve things, things must change
We are changing things
Therefore, we are improving things.
Before you spend my tax money on changing things, please pause and consider if the change is actually for the better. Or just a change.
