Er når mailen fra de studerende indledes med:
“Kære rigtig flinke gut fra datalab, som vi har glemt hvad hedder. “

When is data anonymous? That is a very good question, and one that is increasingly relevant for my work.

Our datalabs at the University Library of Copenhagen (or whatever our current name is), is beginning to be a success. We have a steady increase in the number of students and researchers from the health sciences. And that triggers a recurring discussion.

Let me begin by noting that our users are very conscious of the issues regarding protecting sensitive information.They use encrypted hardware, secure connections to a degree I have only ever seen amongst people security consciuos enough to border on tinfoil folks. But they are still a bit naive about anonomyzing data.

I have no idea how to anonymize data. And the more I read about it, the less sure I am that it is actually possible. People smarter than me are probably able to figure out something. But I fear that this is a game rather like the DRM-games.

Yes, the studios can encrypt their Bluray discs. But they still need to be able to show the movie on a screen. The disc will have to be decrypted somehow. Otherwise it will just show static. And the data you are working with may be stripped of all identifying information. But there still needs to be information in it. Otherwise it is just useless.

So – I cannot advice our students on how to de-identify patients in a clinical study. But I can tell them horror stories. And I do. These are a few of them.

Netflix and IMDB

The classic story is the de-identification of Netflix users. Netflix has periodically released data on their users. Anonymized of course. Which movies have a given user watched, and how has that user rated them.

Another source about information on what movies a person has watched and rated is IMDB. And that information is not so secret. Let us asume that an unknown person has watched ten obscure movies on Netflix, and given the first five a high rating and the others a low. And that a known person on IMBD has rated the same five obscure movies high, and the other five low. Intuition would suggest that those two persons are the same. Is that a problem?

If you live in an area where being gay is a problem, you might not have problem people knowing that you have watched obscure, but innocent movies on IMDB. But the Netflix data, if linked to you, would reveal that you also have watched Another Gay Movie, Philadelphia and Milk. That might be a problem. I don’t think “Salo” is on Netflix. And I’m not necessarily that embarrassed to admit that I have watched it. But I would probably not want people to know that I have watched it ten times (if I had. Its horrifying). Heres a paper on the case.

Postcodes

A lot of demographic data is released to the public. We want people to know if living in a certain area causes cancer. And we want the underlying data out there, because there is just too much data to analyze, so if we could crowdsource that part of the process, it would be nice. So we anonymize the data, but leave in the postcodes. That might be a problem.

The danish postcode 1301 corresponds to the street “Landgreven”. According to www.krak.dk, 17 persons have an adress there. There might be a bit more. They only register people with a phonenumber. And leaves out people with an unlisted number. But let us assume that there are only those 17 persons. 8 of them are women. So if we have health data on medical procedures – broken down by postcode and gender, we might be able to say that one of 8 named women had an abortion. Not that there is much stigma associated with that in Denmark, or at least there shouldnt be. But it is still something you probably would like to keep to yourself.

Twitter, Flickr and graphs

Some people like to be anonomous on Twitter. Looking at the name-calling, flamewars and general pouring crap over people you disagree with on Twitter, it is surprising that not more people are trying to be anonymous on Twitter. But some people have legitimate reasons to try to be anonymous. Whistleblowers, human rigts activists etc.

Social media are characterized by graphs. Not pie charts and such. But networks. Each person is a node, and each connection, following eg, between nodes is an edge. The network defined by nodes and edges is called a graph. Two researchers Narayanan and Shmatikov have made an interesting study, “De-anonymizing social networks”. Take a lot of persons that have accounts on both Twitter and Flickr. Anonymise the Twitter accounts. One third of those Twitter accounts can be linked to the Flickr acocunt of the same person. In spite of the anonymisation.

How? Well, the graph describing who you follow and who follows you on Twitter, will share characteristics with the graph on Flickr. And those graphs are pretty unique. Read more here.

 

At være nøjagtig lige så pisseligeglad med hvad som helst som alle andre åbenbart er.

“Vi skal have annonceret kurser på webben”. Fem mand høj. Vi ved at det skal gøres. Alle ved det skal gøres. Alle kan gøre det. Men der er ingen der gør noget som helst. Jeg ender med at gøre det.

“Regnskabet skal revideres”. Hele bestyrelsen ved at det skal gøres. Kassereren ved det skal gøres. Ingen gør noget som helst. Revisoren – det er mig – ender med at sidde med det i sidste øjeblik. Det er ellers ikke noget der burde komme som en overraskelse. Det skal gøres en gang om året…

Der er simpelthen for meget i mit liv der kun sker hvis jeg tager initiativet. Det er åbenbart kun mig der mener det er vigtigt. Jeg skal simpelthen lære at være nøjagtig ligeså pissehamrende ligeglad med hvad som helst som alle andre omkring mig tydeligvis er. Det vil formentlig være godt for mit blodtryk.

Seneste WordPress opdatering introducerede en fin ny editor. Som angiveligt er helt vildt fantastisk, det nye sort, det de unge vil have og 100% hipster-kompatibel.

Jeg hader den. Denne plugin:

Classic Editor

løser problemet, og giver dig en editor der ligner alle andre editors. I stedet for et eller andet fancy-pancy.

I have a love-hate relationship with visions and missions. The ones that companies and organizations spend a lot of time and ressources on developing. They often have a rather formulaic form:

We exist to restore intellectual capital whilst continuing to collaboratively coordinate information.

This is actually a mission statement from a mission statement generator.

I do love the idea of missions, visions and strategies. It speaks to the engineer in me. We should define the goal we want to achieve, and then break that goal down into individual work-packages. When we have completed all of them, we have achieved our goal. The logical framework approach is a good example.

I also like values. I need consistency. Or, I don’t need it, but it is important to me. I like people and organizations to actually be consistent in their actions. Walk the talk! If you want a work environment that does not discriminate – do not discriminate. Anyone. In any way, I do not really mind that you discriminate based on gender. Go ahead. Just be honest about it. In reality, I will of course hate you if you discriminate based on gender. But the hate will take a more deep and incandecent nature if you discriminate based on gender, while claiming that you are all about equality.

So – I love strategies. I love visions. I love missions. I love values.

On the other hand. I hate them. Most of them are exactly like the example above. An example that is taken from the mission statement generator. Noone is able to explain the difference between mission and vision, and the values all falls for the negation test. And tend to end up being rather tautological. Often they are self contradictory. You can have loyalty in all situations. Or you can have honesty in all situations. But you can’t have both. Sometimes being loyal means holding back on honesty a bit. How do you prioritize your values? I have never seen a description of a hierarchy.

Usually it doesn’t matter. Most values in organizations tend to be nothing more than hot air. And as hot air, easy to dismiss when it is opportune. Try it yourself. Tell your boss that her decision is wrong, because it is in conflict the the defined values of the organization.

So – I love values. But not when they are meaningless.

And if they are – don’t bother defining them. They are just going to be a waste of time.

Et virkeligt seriøst tillidsbrud tager lang tid at komme sig over.

Der skete noget for ca. 2½ år siden. Og det har gentagne gange forbløffet mig hvor lang tid det har taget at komme over det. I fredags oplevede jeg at der var noget selvværd der var vendt tilbage.

Det var rart.

Ja, det er ikke let. Eller. Det er utroligt let.

Som hvid, cis-kønnet, hetero-præsenterende, midaldrende, akademikermand i arbejde fra den vestlige verden, er jeg utroligt priviligeret. Helt vildt priviligeret. Lønmæssigt er jeg blandt de 0.1% bedst stillede i verden. Og det er jeg selvom jeg er statsansat.

På den anden side, er det ikke specielt let. Som hvid, cis-kønnet midaldrende mand, er jeg nemlig også ansvarlig for alt dårligt i verden. Fattigdom i Afrika? Det skyldes at britene udnyttede de stakkels afro-amerikanere som slaver. Og briterne var hvide cis-kønnede, midaldrende mænd, og derfor er jeg medansvarlig for en slavehandel der fandt sted for 300 år siden. Den amerikanske præsident taler grimt om mexicanere. Og han er hvid og cis-kønnet. Det er jeg også. Og derfor er jeg, selvom jeg er dansk statsborger, og har tilbragt sammenlagt lidt under 2 måneder i USA, medskyldig.

Og kvinder! De er undertrykt. Sygeplejersker får ikke lige så meget i løn som ingeniører. Og ingeniører er mænd, mens sygeplejersker er kvinder. Og det er kvindeundertrykkende. Det er også kvinder der står for det meste af husarbejdet. Og eftersom jeg er mand, er det noget jeg er ansvarlig for. Også selvom det ikke er mig der har valgt at blive sygeplejerske. Og selvom det er mig der står for rengøring, oprydning og madlavning herhjemme (fair nok, jeg er gift med en mand. Det vil altid være en mand der står for det praktiske her i lejligheden).

Og det er selvfølgelig bare klynk. Det er et udtryk for en skrøbelig maskulinitet at man som mand gør opmærksom på at det måske ikke er ens skyld at en kvinde har valgt at læse nordisk filologi, og nu får mindre i løn end en mandlig forsikringsaktuar. Og hvis ikke det er udtryk for en skrøbelig maskulinitet, så er den formentlig toksisk. Fordi det at være mand er noget giftigt noget, der ødelægger ting. Man skal ikke som mand klynke over den slags ting. Det skal man tage som en mand. Nemlig. Og samtidig skal man være i kontakt med sine følelser og sådan noget. Men altså ikke på en måde hvor man kommer til at gøre opmærksom på at det egentlig sårer en at blive holdt ansvarlig for andres ulykke. Når man nu faktisk ikke har noget som helst med den at gøre.

Det kan også være lidt belastende at få at vide at samfundet lissom er indrettet til fordel for mænd. Specielt hvis man kommer til at bemærke, at der er tre steder i lovgivningen hvor der gøres forskel på mænd og kvinder. Mænd har værnepligt. Det vil sige at de har pligt til at lade sig slå ihjel for landets forsvar. Kvinder har ret. Hvis de beslutter sig for at de alligevel ikke har lyst, tager de bare hjem. Hvis man som mand nægter, bliver man hentet af politiet. Så er der sociallovgivningen. Kommunerne er forpligtet til at etablere tilbud til udsatte kvinder. Er du udsat mand, må du håbe at kommunen har en ledig bænk du kan sove på. Og endelig har du som mand, skulle du blive far, ret til mindre barsel end barnets mor.

Og skulle du gøre opmærksom på den slags? Så er du privilegieblind. Du er toxisk. Du klynker. Og dine oplevelser er pr. definition irrelevante.

Så. Rant delvist over. Man bør ikke blive overrasket hvis der er mænd der melder sig ud af det show. Langt hen ad vejen orker jeg ikke kønsdebatter. Mine oplevelser er fra start dømt ude. Fordi jeg er mand.

Not that advanced, but I wanted to play around a bit with plotting the raw data from Openstreetmap.

We’re going to Florence this fall. It’s been five years since we last visited the fair city, that has played such an important role in western history.

Openstreetmaps is, as the name implies, open.

I’m going to need some libraries

#library(OpenStreetMap)
library(osmar)
library(ggplot2)
library(broom)
library(geosphere)
library(dplyr)

osmar provides functions to interact with Openstreetmap. ggplot2 is used for the plots, broom for making some objects tidy and dplyr for manipulating data.

Getting the raw data, requires me to define a boundary box, encompassing the part of Florence I would like to work with. Looking at https://www.openstreetmap.org/export#map=13/43.7715/11.2717, I choose these coordinates:

top <- 43.7770
bottom <- 43.7642
left <- 11.2443
right <- 11.2661

After that, I can define the bounding box, tell the osmar functions at what URL we can find the relevant API (this is just the default). And then I can retrieve the data via get_osm(). I immediately save it to disc. This takes some time to download, and there is no reason to do that more than once.

box <- corner_bbox(left, bottom, right, top)
src <- osmsource_api(url = "https://api.openstreetmap.org/api/0.6/")
florence <- get_osm(box, source=src)
saveRDS(florence, "florence.rda")

Lets begin by making a quick plot:

plot(florence, xlim=c(left,right),ylim=c(bottom,top) )

Note that what we get a plot of, is, among other things, of all lines that are partly in the box. If a line extends beyond the box, we get it as well.

Looking at the data:

summary(florence$ways)

## osmar$ways object
## 6707 ways, 9689 tags, 59052 refs 
## 
## ..$attrs data.frame: 
##     id, visible, timestamp, version, changeset, user, uid 
## ..$tags data.frame: 
##     id, k, v 
## ..$refs data.frame: 
##     id, ref 
##  
## Key-Value contingency table:
##         Key         Value Freq
## 1  building           yes 4157
## 2    oneway           yes  456
## 3   highway    pedestrian  335
## 4   highway   residential  317
## 5   bicycle           yes  316
## 6       psv           yes  122
## 7   highway  unclassified  108
## 8   highway       footway  101
## 9   barrier          wall   98
## 10  surface paving_stones   87

I would like to plot the roads and buildings. For some reason there are a lot of highways, of a kind I would probably not call highways.

Anyway, lets make a list of tags. tags() finds the elements that have a key in the tag_list, way finds the lines that are represented by these elements, and find, finds the ID of the objects in “florence” matching this.
find_down() finds all the elements related to these id’s. And finally we take the subset of the large florence data-set, which have id’s matching the id’s we have in from before.

tag_list <- c("highway", "bicycle", "oneway", "building")
dat <- find(florence, way(tags(k %in% tag_list)))
dat <- find_down(florence, way(dat))
dat <- subset(florence, ids = dat)

Now, in a couple of lines, I’m gonna tidy the data. That removes the information of the type of line. As I would like to be able to color highways differently from buildings, I need to keep the information.
Saving the key-part of the tags, and the id:

types <- data.frame(dat$ways$tags$k, dat$ways$tags$id)
names(types) <- c("type", "id")

This gives me all the key-parts of all the tags. And I’m only interested in a subset of them:

types <- types %>% 
  filter(type %in% tag_list)

types$id <- as.character(types$id)

Next as_sp() converts the osmar object to a spatial object (just taking the lines):

dat <- as_sp(dat, "lines")

tidy (from the library broom), converts it to a tidy tibble

dat <- tidy(dat)

That tibble is missing the types – those are added.

new_df <- left_join(dat, types, by="id")

And now we can plot:

new_df %>% 
  ggplot(aes(x=long, y=lat, group=group)) +
  geom_path(aes(color=type)) +
  scale_color_brewer() +
    xlim(left,right) +
  ylim(bottom,top) +
  theme_void() +
theme(legend.position="none")

Nice.

Whats next? Someting like what is on this page: https://github.com/ropensci/osmplotr

Project Euler 187. Semiprimes

I had to look it up. Semiprimes are numbers that are the product of two prime numbers. And only two, although they may be equal.

There are ten of them below 30: 4, 6, 9, 10, 14, 15, 21, 22, 25 and 26.

16 is not. The only primefactor is 2, but it occurs four times.

How many of these semiprimes are there below 10^8?

That should be pretty straightforward: Generate all primes below 10^8, make all the multiplications, and count how many uniqe numbers there are below n, where n=10^8.

One problem:

n <- 10**8
numbers <- primes(n)
length(numbers)

## [1] 5761455

That is a lot of numbers to multiply.

A trick: 2 times all the primes below n/2 will give all the semiprimes that have 2 as one of the primefactors (smaller than n).

3 times all the primes below n/3 will in the same way give all the semiprimes, that have 3 as one of the primefactors.

If I can figure out how many primes there are below n/2, I get the number of semiprimes that has 2 as one of the two primefactors. The same for the number of primes below n/3. If continue that to \(\sqrt(n)\), and add them all together, I should get the total number of semiprimes below n.

One issue though. The first prime below n/2 that I multiply by 2, is 3. And the first prime below n/3 that I multiply by 3 is 2. Both giving 6. I need to figure out how to only count 6 one time.

I just generated all the primes below n. The number of primes below n/2 is:

length(numbers[numbers<n/2])

## [1] 3001134

And the number of primes below n/3 is:

length(numbers[numbers<n/3])

## [1] 2050943

I do want to multiply 3 by 3. But I need to exclude 2.

length(numbers[numbers<n/3 & numbers>=3])

## [1] 2050942

Qap’la! One less prime.

I just need to do it for all of them.

n_semi_primes <- function(x){
  counter <- 0
  for(i in numbers[numbers<=sqrt(x)]){
    counter <- counter + length(numbers[numbers<x/i & numbers>=i])
  }
  return(counter)
}

I’m writing this as a function, taking a limit x. A counter is set to 0. And for all primes i less than \(\sqrt(n)\), I add the number of primes between i and < x/i.

I can test it on the example given:

n_semi_primes(30)

## [1] 10

That was the number of semiprimes below 30. And then it is just a question of running it on 10^8:

answer <- n_semi_primes(10**8)

Euler problem 62

The cube, 41063625 (345^3), can be permuted to produce two other cubes: 56623104 (384³⁾ and 66430125 (405^3). In fact, 41063625 is the smallest cube which has exactly three permutations of its digits which are also cube.

Find the smallest cube for which exactly five permutations of its digits are cube.

Alright. I need to find five cubes, that are permutations of the same digits.

How to check if two numbers are permutations of each other?

We can generate the largest permutation of a given number. If the largest permutation of two numbers are identical, the two numbers are permutations of each other.

So I need a function, that returns the largest permutation of a number. It would be nice, if that function was vectorized.

max_perm <- function(t){
  require(magrittr)
  options(scipen=5)
  t %>% 
    as.character() %>% 
    str_split("") %>% 
    lapply(sort, decreasing=TRUE) %>% 
    lapply(paste0, collapse="") %>% 
    unlist() %>% 
    as.double()
}

Convert the input to character. Split at “”. That returns a list with vectors containing the individual digits of the input. lapply sorts the individual vectors in the list in decreasing order. Then lapply pastes the elements in each vector together with paste0 and “” as the separator. Then it is unlisted, and returned as numeric.

What is worth noting is a thing I was struggling with for far too long. R likes to write numbers in scientific notation. As in “1e+06”. I have not studied the phenomenon in detail. But options(scipen=5) solves the problem. It is the “penalty” used to decide when a number should be written in scientific notation. Unless I change that (trial and error, but it should be larger than whatever is default), as.character(1000000) will return “1e+06”. And the permutations of “1” “e” “+” “0” “6” are not terribly useful in this context.

I’m hazarding a guess that I don’t need to handle cubes of values of more than four digits.

Beginning with a vector of all numbers from 1 to 9999, I convert it to a dataframe. I transmute the first column to a column with the name x.
Then I mutate a second column, cube, into existence, and calculate it as the cube of the x-value. A third column, max_cube, is mutated with the result from my max_perm function above. And tha column is immediately used to group the data, so I get date grouped by identical maximum values of the permutations. I filter on the count of those groups, and only keep the groups that contain 5 elements. Then I ungroup it, and select just the cube column.

I now have a data frame with a single column containing 10 values. They are all cubes, five of them are permutations of each other. The other five are also permutaions of each other. And now I just have to take the smallest of them.

result <- 1:9999 %>% 
  as.double() %>% 
  as.data.frame() %>% 
  transmute(., x = .) %>% 
  mutate(cube = x**3) %>% 
  mutate(max_cube = max_perm(cube)) %>% 
  group_by(max_cube) %>% 
  filter(n()==5) %>% 
  ungroup() %>% 
  select(cube) %>% 
  min()

Before I print the result, so I can enter it into Project Euler, I set options(digits=17).

Done! A good exercise. And a valuable lesson in the importance of the options in R.