This one is going to be quick, we will use the xsitemap package which crawls XML sitemap

library(xsitemap)
library(urltools)
library(XML)
library(httr)
upload <- xsitemapGet("https://www.rforseo.com/sitemap.xml")

## Reaching for XML sitemap... https://www.rforseo.com/sitemap.xml

## regular sitemap detected -  39  web page url(s) found

## ......................................

head(upload)

##                                                             loc    lastmod
## 1                                      https://www.rforseo.com/ 2022-07-29
## 2                  https://www.rforseo.com/classic-r-operations 2021-05-08
## 3                                 https://www.rforseo.com/intro 2022-02-17
## 4                               https://www.rforseo.com/r-intro 2022-08-18
## 5                           https://www.rforseo.com/rpivottable 2022-08-18
## 6 https://www.rforseo.com/analysis/count-words-n-grams-shingles 2021-04-06

step 2 : Launching the URL Inspection API in parallel

We use the parallel package to allow us to run several requests at the same time.

Warning, with regards to the URL Inspection API, the quota is enforced per Search Console website property (calls querying the same site)

I could be useful to create some extra properties using url directories

library(searchConsoleR)
library(lubridate)
library(parallel)
scr_auth()


res <- mclapply(1:nrow(upload), function(i) {
  cat(".")         
  url <-  upload[i,"loc"]
  result <- inspection(url, siteUrl = "sc-domain:rforseo.com", languageCode = NULL)
  
  text <- paste0(url,"§",
                 result[["indexStatusResult"]][["verdict"]],"§",
                 result[["indexStatusResult"]][["coverageState"]],"§",
                 result[["indexStatusResult"]][["robotsTxtState"]],"§",
                 result[["indexStatusResult"]][["indexingState"]],"§",
                 now())
  text
  
  
   }, mc.cores = detectCores())      ## Split this job across 10 cores

res <- data.frame(unlist(res))

library(stringr)

res[,c("url", "verdict", "coverageState", "robotsTxtState", "indexingState", "date")] <- str_split_fixed(res$unlist.r., '§', 6)
res$unlist.r. <- NULL

step 3 : Save data frame inside a Google Cloud Storage bucket

# Load the package
library(googleCloudStorageR)
library(bigQueryR)


## project id
gcs_global_bucket("mindful-path-205008")

gcs_auth()

## custom upload function to ignore quotes and column headers
f <- function(input, output) {
  write.table(input, sep = ",", col.names = FALSE, row.names = FALSE, 
              quote = FALSE, file = output, qmethod = "double")}

## upload files to Google Cloud Storage
gcs_upload(res, name = "res.csv", object_function = f,bucket = "gsc_backup")

I’ve made app that you migh find useful

CTR by Average Position

The first app computes Google Search Queries CTR by Average Position.

👉 https://gokam.shinyapps.io/ctr_pos/

With a big website it looks like this

app code is open sourced here

Crawl Recursively XML sitemaps

The second app help to detects, Crawl and Download XML sitemaps

👉 https://gokam.shinyapps.io/xsitemap/

this app rely primarily on the xsitemap package

This tutorial is relying on a package called Rcrawler by Salim Khalil. It’s a very handy crawler with some nice native functionalities.

After R is being installed and rstudio launched, same as always, we’ll install and load our package:

# install to be run once
install.packages("Rcrawler")
# and loading
library(Rcrawler)

Crawl an entire website with Rcrawler

To launch a simple website analysis, you only need this line of code:

Rcrawler(Website = "https://www.gokam.co.uk/")

It will crawl the entire website and provide you with the data

After the crawl is being done, you’ll have access to:

The INDEX variable

it’s a data frame, if don’t know what’s a data frame, it’s like an excel file. Please note that it will be overwritten every time so export it if you want to keep it!

To take a look at it, just run

View(INDEX)

Most of the columns are self-explanatory. Usually, the most interesting ones are ‘Http Resp‘ and ‘Level‘

The Level is what SEOs call “crawl depth” or “page depth”. With it, you can easily check how far from the homepage some webpages are.

Quick example with BrightonSEO website, let’s do a quick ‘ggplot’ and we’ll be able to see pages distribution by level.

#here the code to run to see the plot

# install ggplot plot library to be run once
install.packages("ggplot2")
# Loading library
library(ggplot2)
# Convert Level to number
INDEX$Level <- as.integer(INDEX$Level)

# Make plot
# 1 define dimensions (only 'Level')
# 2 set up the plot type
# 3 customise the x scale, easier to read
ggplot(INDEX, aes(x=Level))+
       geom_bar()+
       scale_x_continuous(breaks=c(1:10))


#alternative command to count webpages per Level
table(INDEX$Level)

# Should display something like that:
# 0  1   2  3   4   5  6  7  8   9  10
# 1 32 306 91 116 127 61 54 90 149 255

HTML Files

By default, the rcrawler function also store HTML files in your ‘working directory’. Update location by running setwd() function

Let’s go deeper into options by replying to the most commons questions:

So how to extract metadata while crawling?

It’s possible to extract any elements from webpages, using a CSS or XPath selector. We’ll have to use 2 new parameters

• PatternsNames to name the new parameters
• ExtractXpathPat or ExtractCSSPat to setup where to grab it in the web page

Let’s take an example:

#what we want to extract
CustomLabels <- c("title",
                 "h1",
                 "canonical tag",
                 "meta robots",
                 "hreflang",
                 "body class")

# How to grab it
 CustomXPaths <- c("///title",
           "///h1",
           "//link[@rel='canonical']/@href",
           "//meta[@rel='robots']/@content",
           "//link[@rel='alternate']/@hreflang",
           "//body/@class")

 Rcrawler(Website = "https://www.brightonseo.com/",
       ExtractXpathPat = CustomXPaths, PatternsNames = CustomLabels)

You can access the scraped data in two ways:

• option 1 = DATA – it’s an environment variable that you can directly access using the console. A small warning, it’s a ‘list’ a little less easy to read

If you want to convert it to a data frame, easier to deal with, here the code:

NEWDATA <- data.frame(matrix(unlist(DATA), nrow=length(DATA), byrow=T))

• option 2 = extracted_data.csv
  
  It’s a CSV file that has been saved inside your working directory along with the HTML files.

It might be useful to merge INDEX and NEWDATA files, here the code

MERGED <- cbind(INDEX,NEWDATA)

As an example, let’s try to collect webpage type using scraped body class

Let’s extract the first word and feed it inside a new column

MERGED$pagetype <- str_split_fixed(MERGED$X7, " ", 2)[,1]

A little bit a cleaning to make the labels easier to read

MERGED$pagetype_short <- str_replace(MERGED$pagetype, "-default", "")
 MERGED$pagetype_short <- str_replace(MERGED$pagetype_short, "-template", "")
#it's basically deleting "-default" and "-template" from strings as it doesn't help that much understanding data

And then a quick ggplot

library(ggplot2)
p <- ggplot(MERGED, aes(x=Level, fill=pagetype_short))+
   geom_histogram(stat="count")+
   scale_x_continuous(breaks=c(1:10))
p

Want to see something even cooler?

#install package plotly the first time
#install.packages("plotly")
 library(plotly)
 ggplotly(p, tooltip = c("count","pagetype_short"))

This is a static HTML file that can be store anywhere, even on my shared hosting

Explore Crawled Data with rpivottable

#install package rpivottable the first time
#install.packages("rpivottable")
# And loading
 library(rpivottable)
# launch tool 
rpivotTable(MERGED)

This create a drag & drop pivot explorer

It’s also possible make some quick data viz

Full DEMO – see by yourself

Extract more data without having to recrawl

All the HTML files are in your hard drive, so if you need more data extracted, it’s entirely possible.

You can list of your recent crawl by using ListProjects() function,

First, we’re going to load the crawling project HTML files:

LastHTMLDATA <- LoadHTMLFiles("gokam.co.uk-242115", type = "vector")
# or to simply grab the last one:
LastHTMLDATA <- LoadHTMLFiles(ListProjects()[1], type = "vector")

LastHTMLDATA <- as.data.frame(LastHTMLDATA)
colnames(LastHTMLDATA) <- 'html'
LastHTMLDATA$html <- as.character(LastHTMLDATA$html)

Let’s say you forgot to grab h2’s and h3’s you can extract them again using the ContentScraper() also included inside rcrawler package.

for(i in 1:nrow(LastHTMLDATA)) {
   LastHTMLDATA$title[i] <- ContentScraper(HTmlText = LastHTMLDATA$html[i] ,XpathPatterns = "//title")
   LastHTMLDATA$h1[i] <- ContentScraper(HTmlText = LastHTMLDATA$html[i] ,XpathPatterns = "//h1")
   LastHTMLDATA$h2[i] <- ContentScraper(HTmlText = LastHTMLDATA$html[i] ,XpathPatterns = "//h2")
   LastHTMLDATA$h3[i] <- ContentScraper(HTmlText = LastHTMLDATA$html[i] ,XpathPatterns = "//h3")
 }

Categorize URLs using Regex

For those not afraid of regex, here is a complimentary script to categorize URLs. Be careful the regex order is important, some values can overwrite others. Usually, it’s a good idea to place the home page last

# define a default category

INDEX$UrlCat <- "Not match"

 

# create category name

category_name <- c("Category", "Dates", "author page", "Home page")

 
# create category regex, must be the same length

category_regex <- c("category", "2019", "author","example\.com.\/$")

 

# categorize

for(i in 1:length(category_name)){

# display a dot to show the progress
  cat(".")
# run regex test and update value if it matches
# otherwise leave the previous value
  INDEX$UrlCat <- ifelse(grepl(category_regex[i], INDEX$Url, ignore.case = T), category_name[i], INDEX$UrlCat)

}


# View variable to debug

View(INDEX)

What if I want to follow robots.txt rules?

just had Obeyrobots parameter

#like that
Rcrawler(Website = "https://www.gokam.co.uk/", Obeyrobots = TRUE)

What if I want to limit crawling speed?

By default, this crawler is rather quick and can grab a lot of webpage in no times. To every advantage an inconvenience, it’s fairly easy to wrongly detected as a DOS. To limit the risks, I suggest you use the parameter RequestsDelay. it’s the time interval between each round of parallel HTTP requests, in seconds. Example

# this will add a 10 secondes delay between
Rcrawler(Website = "https://www.example.com/", RequestsDelay=10)

Other interesting limitation options:

no_cores: specify the number of clusters (logical cpu) for parallel crawling, by default it’s the numbers of available cores.

no_conn: it’s the number of concurrent connections per one core, by default it takes the same value of no_cores.

What if I want to crawl only a subfolder?

2 parameters help you do that. crawlUrlfilter will limit the crawl, dataUrlfilter will tell from which URLs data should be extracted

Rcrawler(Website = "http://www.glofile.com/sport/", dataUrlfilter ="/sport/", crawlUrlfilter="/sport/" )

How to change user-agent?

#as simply as that
Rcrawler(Website = "http://www.example.com/", Useragent="Mozilla 3.11")

What if my IP is banned?

option 1: Use a VPN on your computer

Option 2: use a proxy

Use the httr package to set up a proxy and use it

# create proxy configuration
proxy <- httr::use_proxy("190.90.100.205",41000)
# use proxy configuration
Rcrawler(Website = "https://www.gokam.co.uk/", use_proxy = proxy)

Where to find proxy? It’s been a while I didn’t need one so I don’t know.

Where are the internal Links?

By default, RCrawler doesn’t save internal links, you have to ask for them explicitly by using NetworkData option, like that:

Rcrawler(Website = "https://www.gokam.co.uk/",  NetworkData = TRUE)

Then you’ll have two new variables available at the end of the crawling:

• NetwIndex var that is simply all the webpage URLs. The row number are the same than locally stored HTML files, so
  row n°1 = homepage = 1.html

• NetwEdges with all the links. It’s a bit confusing so let me explain:

Each row is a link. From and To columns indicate “from” which page “to” which page are each link.

On the image above:
row n°1 is a link from homepage (page n°1) to homepage
row n°2 is a link from homepage to webpage n°2. According to NetwIndex variable, page n°2 is the article about rvest.
etc…

Weight is the Depth level where the link connection has been discovered. All the first rows are from the homepage so Level 0.

Type is either 1 for internal hyperlinks or 2 for external hyperlinks

Count Links

I guess you guys are interested in counting links. Here is the code to do it. I won’t go into too many explanations, it would be too long. if you are interested (and motivated) go and check out the dplyr package and specifically Data Wrangling functions

Count outbound links

count_from <- NetwEdges[,1:2] %>%
#grabing the first two columns
     distinct() %>%
# if there are several links from and to the same page, the duplicat will be removed.
     group_by(From) %>%
     summarise(n = n()) 
# the counting
View(count_from)
# we want to view the results

To make it more readable let’s replace page IDs with URLs

count_from$To <- NetwIndex
View(count_from)

Count inbound links

The same thing but the other way around

count_to -> NetwEdges[,1:2] %>%
#grabing the first two columns
     distinct() %>%
# if there are several links from and to the same page, the duplicat will be removed.
     group_by(To) %>%
     summarise(n = n())
# the counting
View(count_to)

# we want to view the results

Again to make it more readable

count_to$To <- NetwIndex
View(count_to)

So the useless ‘author page‘ has 14 links pointing at it, as many as the homepage… Maybe I should fix this one day.

Compute ‘Internal Page Rank’

Many SEOs, I spoke to, seem to be very interested in this. I might as well add here the tutorial. It is very much an adaptation of Paul Shapiro awesome Script.

But Instead of using ScreamingFrog export file, we will use the previously extracted links.

links <- NetwEdges[,1:2] %>%
   #grabing the first two columns
   distinct() 
# loading igraph package
 library(igraph)
# Loading website internal links inside a graph object
 g <- graph.data.frame(links)

# this is the main function, don't ask how it works
 pr <- page.rank(g, algo = "prpack", vids = V(g), directed = TRUE, damping = 0.85)

# grabing result inside a dedicated data frame
 values <- data.frame(pr$vector)
 values$names <- rownames(values)

# delating row names
 row.names(values) <- NULL

# reordering column
 values <- values[c(2,1)]
# renaming columns
 names(values)[1] <- "url"
 names(values)[2] <- "pr"
 View(values)

Let make it more readable, we’re going to put the number on a ten basis, just like when the PageRank was a thing.

#replacing id with url
values$url <- NetwIndex
# out of 10
 values$pr <- round(values$pr / max(values$pr) * 10)
#display
 View(values)

On 15 webpages website, it’s not very impressive but I encourage you to try on a bigger website.

What if a website is using a JavaScript framework like React or Angular?

RCrawler handly includes Phantom JS, the classic headless browser.
Here is how to to use

# Download and install phantomjs headless browser
# takes 20-30 seconds usually
install_browser()

# start browser process 
br <-run_browser()

After that, reference it as an option

Rcrawler(Website = "https://www.example.com/", Browser = br)

# don't forget to stop browser afterwards
stop_browser(br)

It’s fairly possible to run 2 crawls, one with and one without, and compare the data afterwards

This Browser option can also be used with the other Rcrawler functions.

⚠️ Rendering webpage means every Javascript files will be run, including Web Analytics tags. If you don’t take the necessary precaution, it’ll change your Web Analytics data

So what’s the catch?

Rcrawler is a great tool but it’s far from being perfect. SEO will definitely miss a couple of things like there is no internal dead links report, It doesn’t grab nofollow attributes on Links and there is always a couple of bugs here and there, but overall it’s a great tool to have.

Another concern is the git repo which is quite inactive.

This is it. I hope you did find this article useful, reach to me for slow support, bugs/corrections or ideas for new articles. Take care.

ref:
Khalil, S., & Fakir, M. (2017). RCrawler: An R package for parallel web crawling and scraping. SoftwareX, 6, 98-106.

One way if dealing with that is to export the data, another very classic one is to use a GA filter remove them but I not a big fan of deleting data that might be useful one day.

So now I’m using a calculated direct inside Google Data Studio, here is the formula to copy past for the savvy users:

REGEXP_REPLACE(Landing Page,"(.)\?.","\1")

For the other, here is the step by step:
Open your report and go to “Ressource” -> “Manage added sources”

Choose edit

and ‘ADD A FIELD’ on the top right

Name this new field something memorable and copy post the previous formula and you are good to go.
I hope you will find this useful

Here is a simple demo from the package documentation using the IMDb website:

# Package installation, instruction to be run only once 
install.packages("rvest") 

# Loading rvest package
library(rvest)

The first step is to crawl the URL and store the webpage inside a ‘lego_movie’ variable.

lego_movie <- read_html("http://www.imdb.com/title/tt1490017/")

Quite straightforward, isn’t it?

Beware lego_move is now an xml_document that need to be parse in order to extract the data. Here is how to do it:

rating <- lego_movie %>% 
   html_nodes("strong span") %>%
   html_text() %>%
   as.numeric()

For those who don’t know %>% operator is like the | ( “pipe”) for a terminal command line. The operations are carried out successively. Meaning the results of the previous command are the entries for the next one.

html_nodes() function will extract from our webpage, HTML tags that match CSS style query selector. In this case, we are looking for a <span> tag whose parent is a <strong> tag.
then script will extract the inner text value using html_text() then convert it to a number using as.numeric().

Finally, it will store this value inside rating variable to display the value just write:

rating

# it should display > [1] 7.8

Let’s take another example. This time we are going to grab the movies’ cast.

Having a look at the HTML DOM, it seems that we need to grab an HTML <img> tag who’s parent tag have ‘titleCast’ as an id and ‘primary_photo’ as a class name and then we’ll need to extract the alt attribute

cast <- lego_movie %>%
   html_nodes("#titleCast .primary_photo img") %>%
   html_attr("alt")

 cast

# Should display:
# >  [1] "Will Arnett"     "Elizabeth Banks" "Craig Berry"
# >  [4] "Alison Brie"     "David Burrows"   "Anthony Daniels"
# >  [7] "Charlie Day"     "Amanda Farinos"  "Keith Ferguson"
# > [10] "Will Ferrell"    "Will Forte"      "Dave Franco"
# > [13] "Morgan Freeman"  "Todd Hansen"     "Jonah Hill"

Last example, we want the movie poster url. First step is to grab <img> tag who’s parent have a class name ‘poster’ Then extract src attribute and display it

poster <- lego_movie %>%
   html_nodes(".poster img") %>%
   html_attr("src")

 poster

# Shoudl display:
# [1] "https://m.media-amazon.com/images/M/MV5BMTg4MDk1ODExN15BMl5BanBnXkFtZTgwNzIyNjg3MDE@.V1_UX182_CR0,0,182,268_AL.jpg"

Now a real-life crawl example

Now that we’ve seen an example by the book. We’ll switch to something more useful and a little bit more complex. Using the following tutorial, you’ll be able to extract the review score of any WordPress plugins over time.

For example here are the stats for Yoast, the famous SEO plugin:

Here are the ones’ for All in one SEO, his competitor

Very useful to follow if your favourite plugin new release is well received or not.

But before that, a little warning, the source code I’m about to show you has been made by me. It’s full of flaws, couple of stack overflow copypasta but… it works. 😅 So Dear practitioners please don’t judge me
It’s one of the beauties of R, you get your ends relatively easily.

(but I gladly accept any ideas to make this code easier for beginner, don’t hesitate to contact me)

So let’s get to it, the first step is to grab a reviews page URL. On this one, we have 49 pages of reviews.

We’ll have to make a loop to run into each pagination. Another problem is that no dates are being displayed but only durations, so we’ll have to convert them.

As usual, we’ll first load the necessary packages. If there are not installed yet, run the install.packages() function as seen before.

#Loading packages
library(tidyverse)
library(rvest)

# we store plugin url inside a variable, to make the code easy to reuse
pluginurl <- "https://wordpress.org/support/plugin/wp-fastest-cache/"

# we create and empty dataframe to receive the data that will be retrieved from each pagination. If you don't know what's a data frame think of them as excel file
all_reviews <- data.frame()

#####   beginning of the LOOP ####
# if copy past stuff, don't forget to grab the code until the end of the loop at least
for(i in 1:49) {

# sending to console the loop status
# paste0() function is just a concatenation function with a weird name
message(paste0("Page ",i))

# faculative: make a small break betweeh each loop iteration
# this pause the loop for 2 secondes
# Sys.sleep(2)

# we grab the webpage and store the result inside html_page variable to be able to reuse it several times
 html_page <- read_html(paste0(pluginurl,"reviews/page/",i,"/")) 

# html_nodes is function that use the css or xpath to extract the value from the html page. This part is to extract the number of stars
 reviews <- html_nodes(html_page, ".wporg-ratings")

If you need help to select elements, chrome inspector is great. You can copy/paste xpath and .css style selector directly:

# Then we are getting every htmml attributes values into columns and rows
# it's a copy/past from stackoverflow, it's works don't ask me how.
 extract <- bind_rows(lapply(xml_attrs(reviews), function(x) data.frame(as.list(x), stringsAsFactors=FALSE)))

In other words, it transforms this HTML data hard to deal with

into a clean data frame with nice columns

# using the extract() function get the number of stars
extract <- extract %>% extract(title,c("note"))

# same process but this time to extract the duration
# Grabing from the html file, the duration being displayed
dates <- html_nodes(html_page, ".bbp-topic-freshness")

# Extracting the real duration value from text: we remove line breaks and what's after "ago"
 extract$dates <- html_text(dates, trim = T) %>%  str_replace_all("[\r|\n|\t]" , "") %>% str_replace_all(" ago.*$" , "")

# apply duration type to values, necessary for future conversions
# more info https://lubridate.tidyverse.org/reference/duration.html
extract$duration <- lubridate::as.duration(extract$dates)


# removing from the data frame the now useless columns & rows
 extract$class <- NULL
 extract$title <- NULL
 extract$style <- NULL
 extract$note$class <-NULL
 extract$note$style <- NULL
 extract <- extract[-1,]

# erase rownames
 rownames(extract) <- c()

# converte values to the right type
 extract$note <- as.vector(extract$note)
 extract$note <- as.numeric(extract$note)

# adding all date retrieved during this loop to the main data frame 'all_reviews' 
 all_reviews <- rbind(all_reviews, extract)   

##### END OF THE LOOP #####

 }

The next step is to convert these durations into days. It’s going to be quick:

# .Data is the number of seconds, we divided by 86400 to have the number of days and we round it
all_reviews$duration2 <- round(all_reviews$duration@.Data/86400)

# Today date minus review age will give us the review date 
all_reviews$day <- today()-all_reviews$duration2

# we want to see number of stars as a category not as a scale
all_reviews$note <- as.factor(all_reviews$note)

the data is now ready, export your data or make a small graph to display it using ggplot package

library(ggplot2)
ggplot(all_reviews, aes(x=day, fill=note))+
   geom_histogram()

this is it. I hope you find it useful. If you have problems, reach to me on twitter maybe I can help

Export your data into a CSV

assuming your data is store inside df var, fairly simple:

#setup where to write the file
setwd("~/Desktop")
# en write the file
write.csv(df, "data.csv")

Export your data into an excel file

A little bit more complex, we’ll use the ‘xlsx’ package

#setup where to write the file
setwd("~/Desktop")

# if the package is not instal yet, run this  
# install.packages("xlsx")

# Loading the package 
library(xlsx)

# we write the file 
write.xlsx(df, "data.xlsx")

A few more tips for you:

I’ll like to use the sheetName option to explicitly name the tab. The default name is “Sheet1”. Quite useful to have a record of when the file has been generated for example. Replace last instruction what follows and you’ll be able to know.

write.xlsx(df, "data.xlsx", sheetName=format(Sys.Date(), "%d %b %Y"))

Another good one that I like is to send the excel file to a Shared folder directly. Replace first instruction by

setwd("/Users/me/Dropbox/Public")

Of course, replace the file path by yours.

Send your data by email

If the data to send it not to big, another interesting idea is to send it by email using the ‘gmailr’ package.

#install.packages("gmailr")
#install.packages("tableHTML")

# Packages loading
library(gmailr)
# This one is usefull to transform a data frame into an HTML <table>
library(tableHTML)

# This will allow you to connect to gmail
# replace the fake value by your key and secret
# more info here: https://gargle.r-lib.org/articles/get-api-credentials.html
gm_auth_configure("mykey.apps.googleusercontent.com", "mysecret")

#transform the data frame 'df' to a html table
msg = tableHTML(df)

# Construct email
test_email <- gm_mime() %>%
              gm_to("another@example.com") %>%
              gm_from("me@example.com") %>%
              gm_subject("Email title") %>%
              gm_html_body(paste("Hi Mate,<br />
Here are the data you requested:", msg,"<br /><br />Kind regards,<br />François"))
# end send it
gm_send_message(test_email)