I’m going to take a look at Acta Chemica Scandinavica. Ideally I would like to harvest the entire journal, including all pdfs. But for now, I’m going to harvest the metadata, and ask nicely about getting access to the pdfs without downloading them automatically.

So. First up, getting the metadata.

Datanalysis progresses in five steps:

• (Harvesting) raw data
• Forcing the data to be technically correct
• Coerce the data to be consistent
• Analyse the data
• Present the interesting results

Step 1, getting the raw data. I am going to cheat a bit, and make sure that the data is technically correct while harvesting.

ActaChemScand, as presented on the webpage, gives us access to one volume at a time. We choose a decade, the 40-ties, 50ties etc, and then the volume in the chosen decade we want to look at.

Choosing volume 1, brings us to this url: http://actachemscand.dk/volume.php?select1=1&vol=1 Choosing volume 2, gives us http://actachemscand.dk/volume.php?select1=1&vol=2 And volume 3: http://actachemscand.dk/volume.php?select1=1&vol=3

Volume 4 is in the fifties. Therefore the url changes: http://actachemscand.dk/volume.php?select1=2&vol=4

That seems pretty straightforward. Change the select1 part of the url, based on the decade, and the vol to the volume we are interested in, and it should be simple to generate af list of urls to harvest. There is a small trick, that makes things easier. The select1 variable is actually not needed. It just provides assistance in navigation. Therefore volume 5 can be accessed through this url: http://actachemscand.dk/volume.php?vol=5 Even easier!

But wait. There are supplements. And in the 70ties, there is introduced to series, A and B. Lets take a look: Choosing the supplemental volume 17 (from the sixties), leads us to this url: http://actachemscand.dk/volume.php?select1=3&vol=1017

29A from 1975 is: http://actachemscand.dk/volume.php?select1=4&vol=290 And 29B is: http://actachemscand.dk/volume.php?select1=4&vol=291

So. Not that simple after all. There is a system: Volumes 1 through 27 is just vol=1, vol=2 . vol=27. And vol 17 supp is vol=1017. After that volume xxA is xx0 and xxb is xx1. As in vol 34A is 340, and vol 34B is 341.

One should be able to write a script, that handled that. However, I figured it was easier to list it all in Excel, and construct a dataframe with the needed data manually. Maybe because I did this rather late in the day, and I was out of coffee. So:

url_id <- c(1, 2, 3,4,5,6,7,8,9,10, 11, 12, 13, 14, 15, 16, 17, 1017, 18,
            19, 20, 21, 22, 23, 24, 25, 26, 27, 280, 281, 290, 291, 300, 301, 
            310, 311, 320, 321, 330, 331, 340, 341, 350, 351, 360, 361, 370, 371, 380, 
            381, 390, 391, 400, 401, 410, 411, 420, 421, 430, 440, 450, 460, 470, 480, 
            490, 500, 510, 520, 530)
year <- c(1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957,  
          1958, 1959, 1960, 1961, 1962, 1963, 1963,1964, 1965, 1966, 1967, 1968, 
          1969, 1970, 1971, 1972, 1973, 1974, 1974, 1975, 1975, 1976, 1976, 1977, 1977, 
          1978, 1978, 1979, 1979, 1980, 1980, 1981, 1981, 1982, 1982, 1983, 1983, 1984,
          1984 ,1985 ,1985, 1986, 1986, 1987, 1987, 1988, 1988, 1989, 1990, 1991, 1992, 
          1993, 1994, 1995, 1996, 1997 ,1998, 1999 )
vol <- c("1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17",
         "17s","18","19","20","21","22","23","24","25","26","27","28a","28b","29a",
         "29b","30a","30b","31a","31b","32a","32b","33a","33b","34a","34b","35a","35b",
         "36a","36b","37a","37b","38a","38b","39a","39b","40a","40b","41a","41b","42a",
         "42b","43","44","45","46","47","48","49","50","51","52","53")

url_id <- c(1, 2)
year <- c(1947, 1948)
vol <- c("1", "2")

acta <- data.frame(url_id, year, vol, stringsAsFactors=FALSE)

Okay. Now I have a dataframe, with three columns: url_id, which is the number I need to attach to a baseurl, to get to the relevant webpage. year, the publication year – not just a continuous list of years, as some years had more than one volume. And vol, the volume number. As strings, as we get to vol 28A after a few years.

Oh, and instead of harvesting everything, I modify it, to only look at the first two years. It takes some time to harvest everything.

We’re going to need some libraries. And we’re going to completely ignore all the warnings:

library(RCurl)

## Warning: package 'RCurl' was built under R version 3.2.3

## Loading required package: bitops

## Warning: package 'bitops' was built under R version 3.2.3

library(XML)

## Warning: package 'XML' was built under R version 3.2.3

library(rvest)

## Warning: package 'rvest' was built under R version 3.2.5

## Loading required package: xml2

## Warning: package 'xml2' was built under R version 3.2.5

## 
## Attaching package: 'rvest'

## The following object is masked from 'package:XML':
## 
##     xml

library(textcat)

## Warning: package 'textcat' was built under R version 3.2.5

• RCurl is used for getting my greasy hands on the webpages.
• XML & rvest is used for parsing and accessing the interesting parts of the data. rvest is the reason I’m writing this up in R Markdown, rather than Jupyter. I could not get rvest to run on the Jupyter platform without crashing the kernel.
• textcat. Well we’ll get back to that in detail, but it’s used to determine, or rather guess, the language of the papers.

I’m not going to show screenshots here. But we need to take a look at the HTML-markup on the pages we’re going to harvest. All the papers are listed in a table, with the css-tag “.resulttable”. Within it, we have individual table cells, with the tag “td“. There is some formatting within the tag, that we are going to use later.

But the point is, that we need to get a list into R of all the “fields” within the “.resulttable” tags.

Lets try that with the first volume.

adress <- "http://actachemscand.dk/volume.php?vol=1"

url <- read_html(adress)
papers <- html_nodes(url, ".resulttable") %>%
  html_nodes("td")

head(papers)

## {xml_nodeset (6)}
## [1] <td>\n<select name="select1" onchange="ldMenu(this.selectedIndex);"  ...
## [2] <td>\n<select name="vol" size="1" style="width: 120px"><option selec ...
## [3] <td>\n<input type="submit" value="Go" alt="Go" /></td>
## [4] <td>\n  <b>Some Improvements in Electrophoresis.</b>\n  <p><a href=" ...
## [5] <td>\n  <b>Di-O-alkylmonothiophosphates and Di-O-alkylmonoselenophos ...
## [6] <td>\n  <b>On the determination of Reducing Sugars.</b>\n  <p><a hre ...

Cool. read_html grabs the content of the adress and stores it in the variable “url”. And then we pass it through html_nodes twice. First we ask to get the content of the tag “.resulttable”. And then we pass the result (thats the %>% part) to html_nodes again, this time asking for the contents of the tags “td”.

The result is a set of html nodes, one for each instance of the td tag on the page. Those can be adresses almost like a list.

Looking at the head of the result, we see that we actually have to go to result number 4, before we get to a paper. For some reason, the navigation is embedded in the .resulttable. A bit annoying, but as long as its the same way on every page, it’s not a problem.

Let me just check.

…

Yep, it’s the same way on every page. Or at least on all the five pages I tested. No problem, we’ll just start the loop at 4 instead of 1.

Nice. Now we can look at papers[4] and get the data out of it.

The print function truncates the output. Thats usually OK, but here, I would like to take a look at all the data. Enter the paste function. But! I’m going to want to output this in pdf. RStudio does this by running everything through LaTeX. And as anyone who has ever done anything with LaTeX knows, you often run into very long lines that does not break. This little helper function solves that problem, sort of. It’s shamelessly stolen from http://stackoverflow.com/questions/24020617/textwrapping-long-string-in-knitr-output-rstudio

str_break = function(x, width = 60L) {
  n = nchar(x)
  if (n <= width) return(x)
  n1 = seq(1L, n, by = width)
  n2 = seq(width, n, by = width)
  if (n %% width != 0) n2 = c(n2, n)
  substring(x, n1, n2)
}

And then:

str_break(paste(papers[4]))

## [1] "<td>\n  <b>Some Improvements in Electrophoresis.</b>\n  <p><a "  
## [2] "href=\"/author.php?aid=81\">Astrup, Tage</a>; <a href=\"/author" 
## [3] ".php?aid=36\">Brodersen, Rolf</a></p>\n  <div id=\"pages\"><b>Pa"
## [4] "ges:</b> 1-7.</div>\n  <br />\n  <div id=\"doi\"><b>DOI number:<"
## [5] "/b> 10.3891/acta.chem.scand.01-0001</div>\n  <div id=\"downloa"  
## [6] "d\"><b>Download as:</b> <a target=\"_blank\" href=\"/pdf/acta_vo"
## [7] "l_01_p0001-0007.pdf\">PDF</a> <a href=\"/djvu/acta_vol_01_p000"  
## [8] "1-0007.djvu\">DjVu</a></div>\n</td>"

Much better.

The title is bold. Thats the “b” tag. So is the text “Pages”, “DOI number:” and “Download as:”. But the title always comes first. The author list is the only part of the reference that is enclosed in “b” tags. Pagenumbers and DOI is enclosed in “div”-tags as the only parts of the reference.

bolds <- html_nodes(papers[4], "b") %>%
  html_text()
title <- bolds[1]
print(title)

## [1] "Some Improvements in Electrophoresis."

authors <- html_nodes(papers[4], "p") %>%
  html_text()
print(authors)

## [1] "Astrup, Tage; Brodersen, Rolf"

pagesAndDOI <- html_nodes(papers[4], "div") %>%
  html_text()

pages <- pagesAndDOI[1]
pages <- substring(pages, 8, nchar(pages)-1)
print(pages)

## [1] "1-7"

doi <- pagesAndDOI[2]
doi <- substring(doi, 13, nchar(doi))
print(doi)

## [1] "10.3891/acta.chem.scand.01-0001"

What did I do there?

I passed papers[4] throuhg the html_nodes funciton, looking for -tags. And passed the result to html_text, which gave me the actual textual content for what was in those tags. That gave me a list (sort of), of all the bold text in papers[4]. The first is the title, so title <- bolds[1].

The authors was the only text that was enclosed in

-tags. Same procedure, only this time there is only one result, so authors contains the right information directly.

Pagenumbers and DOI is in

year <- acta[1,]$year
vol <- acta[1,]$vol

print(year)

## [1] 1947

print(vol)

## [1] "1"

url_list <- papers[4] %>%
  html_nodes("a") %>%
  html_attr("href")
urls <- lapply(url_list, function(ch) grep("pdf", url_list))
url <- url_list[urls[[1]]]
print(url)

## [1] "/pdf/acta_vol_01_p0001-0007.pdf"

language <- textcat(title)
print(language)

## [1] "english"

masterDF <- NULL
masterDF <- data.frame(title = character(0), authors = character(0), language = character(0), pages = character(0), doi = character(0), volume = character(0), year = character(0), url = character(0), stringsAsFactors = F)

getPaperDetails <- function(paper, volid){
  title <- html_nodes(paper, "b") %>%
    html_text()
  title <- title[1]
  authors <- html_nodes(paper, "p") %>%
    html_text()
  pagesAndDOI <- html_nodes(paper, "div") %>%
    html_text()
  pages <- pagesAndDOI[1]
  pages <- substring(pages, 8, nchar(pages)-1)
  doi <- pagesAndDOI[2]
  doi <- substring(doi, 13, nchar(doi))
  language <- textcat(title)
  year <- acta[which(acta$url_id==volid),]$year
  volume <- acta[which(acta$url_id==volid),]$vol
  url_list <- paper %>% html_nodes("a") %>% html_attr("href")
  urls <- lapply(url_list, function(ch) grep("pdf", url_list))
  url <- url_list[urls[[1]]]
  
  result <- list(title=title, authors=authors, language=language, pages=pages, 
                 doi=doi, volume=volume, year=year, url=url)
}

base_url <- "http://actachemscand.dk/volume.php?vol="

for(i in 1:length(acta$url_id)){
  weburl <- paste(base_url,acta$url_id[i], sep="")
  webpage <- read_html(weburl)
  papers <- html_nodes(webpage, ".resulttable") %>%
    html_nodes("td")
  for(j in 4:length(papers)){
    masterDF[nrow(masterDF)+1,] <- getPaperDetails(papers[j],acta$url_id[i])
  }
}

head(masterDF)

##     title
## 1 Some Improvements in Electrophoresis.
## 2 Di-O-alkylmonothiophosphates and Di-O-alkylmonoselenophosphates and the Corresponding Pseudohalogens.
## 3 On the determination of Reducing Sugars.
## 4 On the Application of p-Carboxyphenylhydrazones in the Identification of Carbonyl Compounds.
## 5 A Note on the Growth Promoting Properties of an Enzymatic Hydrolysate of Casein.
## 6 Die Konstitution der Harzphenole und ihre biogenetischen Zusammenhänge. X. Herausspaltung des "MIttelstückes" des Pinoresinols
##                             authors language pages
## 1     Astrup, Tage; Brodersen, Rolf  english   1-7
## 2                        Foss, Olav  english  8-31
## 3    Blom, Jakob; Rosted, Carl Olof  english 32-53
## 4                      Veibel, Stig  english 54-68
## 5                     Ågren, Gunnar  english 69-70
## 6 Erdtman, Holger; Gripenberg, Jarl   german 71-75
##                               doi volume year
## 1 10.3891/acta.chem.scand.01-0001      1 1947
## 2 10.3891/acta.chem.scand.01-0008      1 1947
## 3 10.3891/acta.chem.scand.01-0032      1 1947
## 4 10.3891/acta.chem.scand.01-0054      1 1947
## 5 10.3891/acta.chem.scand.01-0069      1 1947
## 6 10.3891/acta.chem.scand.01-0071      1 1947
##                               url
## 1 /pdf/acta_vol_01_p0001-0007.pdf
## 2 /pdf/acta_vol_01_p0008-0031.pdf
## 3 /pdf/acta_vol_01_p0032-0053.pdf
## 4 /pdf/acta_vol_01_p0054-0068.pdf
## 5 /pdf/acta_vol_01_p0069-0070.pdf
## 6 /pdf/acta_vol_01_p0071-0075.pdf

saveRDS(masterDF, file="data.Rda")