Best practices for API packages
- - - -So you want to write an R client for a web API? This document walks through the key issues involved in writing API wrappers in R. If you’re new to working with web APIs, you may want to start by reading “An introduction to APIs” by zapier.
-Overall design
-APIs vary widely. Before starting to code, it is important to understand how the API you are working with handles important issues so that you can implement a complete and coherent R client for the API.
-The key features of any API are the structure of the requests and the structure of the responses. An HTTP request consists of the following parts:
--
-
- HTTP verb (
GET,POST,DELETE, etc.)
- - The base URL for the API -
- The URL path or endpoint -
- URL query arguments (e.g.,
?foo=bar)
- - Optional headers -
- An optional request body -
An API package needs to be able to generate these components in order to perform the desired API call, which will typically involve some sort of authentication.
-For example, to request that the GitHub API provides a list of all issues for the httr repo, we send an HTTP request that looks like:
--> GET /repos/hadley/httr HTTP/1.1
--> Host: api.github.com
--> Accept: application/vnd.github.v3+jsonHere we’re using a GET request to the host api.github.com. The url is /repos/hadley/httr, and we send an accept header that tells GitHub what sort of data we want.
In response to this request, the API will return an HTTP response that includes:
--
-
- An HTTP status code. -
- Headers, key-value pairs. -
- A body typically consisting of XML, JSON, plain text, HTML, or some kind of binary representation. -
An API client needs to parse these responses, turning API errors into R errors, and return a useful object to the end user. For the previous HTTP request, GitHub returns:
-<- HTTP/1.1 200 OK
-<- Server: GitHub.com
-<- Content-Type: application/json; charset=utf-8
-<- X-RateLimit-Limit: 5000
-<- X-RateLimit-Remaining: 4998
-<- X-RateLimit-Reset: 1459554901
-<-
-<- {
-<- "id": 2756403,
-<- "name": "httr",
-<- "full_name": "hadley/httr",
-<- "owner": {
-<- "login": "hadley",
-<- "id": 4196,
-<- "avatar_url": "https://avatars.githubusercontent.com/u/4196?v=3",
-<- ...
-<- },
-<- "private": false,
-<- "html_url": "https://github.com/hadley/httr",
-<- "description": "httr: a friendly http package for R",
-<- "fork": false,
-<- "url": "https://api.github.com/repos/hadley/httr",
-<- ...
-<- "network_count": 1368,
-<- "subscribers_count": 64
-<- }Designing a good API client requires identifying how each of these API features is used to compose a request and what type of response is expected for each. It’s best practice to insulate the end user from how the API works so they only need to understand how to use an R function, not the details of how APIs work. It’s your job to suffer so that others don’t have to!
-First steps
-Send a simple request
-First, find a simple API endpoint that doesn’t require authentication: this lets you get the basics working before tackling the complexities of authentication. For this example, we’ll use the list of httr issues which requires sending a GET request to repos/hadley/httr:
library(httr)
-github_api <- function(path) {
- url <- modify_url("https://api.github.com", path = path)
- GET(url)
-}
-
-resp <- github_api("/repos/hadley/httr")
-resp
-#> Response [https://api.github.com/repositories/2756403]
-#> Date: 2017-08-18 17:47
-#> Status: 200
-#> Content-Type: application/json; charset=utf-8
-#> Size: 5.71 kB
-#> {
-#> "id": 2756403,
-#> "name": "httr",
-#> "full_name": "r-lib/httr",
-#> "owner": {
-#> "login": "r-lib",
-#> "id": 22618716,
-#> "avatar_url": "https://avatars0.githubusercontent.com/u/22618716?v=4",
-#> "gravatar_id": "",
-#> "url": "https://api.github.com/users/r-lib",
-#> ...Parse the response
-Next, you need to take the response returned by the API and turn it into a useful object. Any API will return an HTTP response that consists of headers and a body. While the response can come in multiple forms (see above), two of the most common structured formats are XML and JSON.
-Note that while most APIs will return only one or the other, some, like the colour lovers API, allow you to choose which one with a url parameter:
-GET("http://www.colourlovers.com/api/color/6B4106?format=xml")
-#> Response [http://www.colourlovers.com/api/color/6B4106?format=xml]
-#> Date: 2017-08-18 17:47
-#> Status: 200
-#> Content-Type: text/xml; charset=utf-8
-#> Size: 970 B
-#> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
-#> <colors numResults="1" totalResults="9853929">
-#> <color>
-#> <id>903893</id>
-#> <title><![CDATA[wet dirt]]></title>
-#> <userName><![CDATA[jessicabrown]]></userName>
-#> <numViews>480</numViews>
-#> <numVotes>1</numVotes>
-#> <numComments>0</numComments>
-#> <numHearts>0</numHearts>
-#> ...
-GET("http://www.colourlovers.com/api/color/6B4106?format=json")
-#> Response [http://www.colourlovers.com/api/color/6B4106?format=json]
-#> Date: 2017-08-18 17:47
-#> Status: 200
-#> Content-Type: application/json; charset=utf-8
-#> Size: 569 BOthers use content negotiation to determine what sort of data to send back. If the API you’re wrapping does this, then you’ll need to include one of accept_json() and accept_xml() in your request.
If you have a choice, choose json: it’s usually much easier to work with than xml.
-Most APIs will return most or all useful information in the response body, which can be accessed using content(). To determine what type of information is returned, you can use http_type()
http_type(resp)
-#> [1] "application/json"I recommend checking that the type is as you expect in your helper function. This will ensure that you get a clear error message if the API changes:
-github_api <- function(path) {
- url <- modify_url("https://api.github.com", path = path)
-
- resp <- GET(url)
- if (http_type(resp) != "application/json") {
- stop("API did not return json", call. = FALSE)
- }
-
- resp
-}NB: some poorly written APIs will say the content is type A, but it will actually be type B. In this case you should complain to the API authors, and until they fix the problem, simply drop the check for content type.
-Next we need to parse the output into an R object. httr provides some default parsers with content(..., as = "auto") but I don’t recommend using them inside a package. Instead it’s better to explicitly parse it yourself:
-
-
- To parse json, use
jsonlitepackage.
- - To parse xml, use the
xml2package.
-
github_api <- function(path) {
- url <- modify_url("https://api.github.com", path = path)
-
- resp <- GET(url)
- if (http_type(resp) != "application/json") {
- stop("API did not return json", call. = FALSE)
- }
-
- jsonlite::fromJSON(content(resp, "text"), simplifyVector = FALSE)
-}Return a helpful object
-Rather than simply returning the response as a list, I think it’s a good practice to make a simple S3 object. That way you can return the response and parsed object, and provide a nice print method. This will make debugging later on much much much more pleasant.
-github_api <- function(path) {
- url <- modify_url("https://api.github.com", path = path)
-
- resp <- GET(url)
- if (http_type(resp) != "application/json") {
- stop("API did not return json", call. = FALSE)
- }
-
- parsed <- jsonlite::fromJSON(content(resp, "text"), simplifyVector = FALSE)
-
- structure(
- list(
- content = parsed,
- path = path,
- response = resp
- ),
- class = "github_api"
- )
-}
-
-print.github_api <- function(x, ...) {
- cat("<GitHub ", x$path, ">\n", sep = "")
- str(x$content)
- invisible(x)
-}
-
-github_api("/users/hadley")
-#> <GitHub /users/hadley>
-#> List of 30
-#> $ login : chr "hadley"
-#> $ id : int 4196
-#> $ avatar_url : chr "https://avatars3.githubusercontent.com/u/4196?v=4"
-#> $ gravatar_id : chr ""
-#> $ url : chr "https://api.github.com/users/hadley"
-#> $ html_url : chr "https://github.com/hadley"
-#> $ followers_url : chr "https://api.github.com/users/hadley/followers"
-#> $ following_url : chr "https://api.github.com/users/hadley/following{/other_user}"
-#> $ gists_url : chr "https://api.github.com/users/hadley/gists{/gist_id}"
-#> $ starred_url : chr "https://api.github.com/users/hadley/starred{/owner}{/repo}"
-#> $ subscriptions_url : chr "https://api.github.com/users/hadley/subscriptions"
-#> $ organizations_url : chr "https://api.github.com/users/hadley/orgs"
-#> $ repos_url : chr "https://api.github.com/users/hadley/repos"
-#> $ events_url : chr "https://api.github.com/users/hadley/events{/privacy}"
-#> $ received_events_url: chr "https://api.github.com/users/hadley/received_events"
-#> $ type : chr "User"
-#> $ site_admin : logi FALSE
-#> $ name : chr "Hadley Wickham"
-#> $ company : chr "@rstudio "
-#> $ blog : chr "http://hadley.nz"
-#> $ location : chr "Houston, TX"
-#> $ email : NULL
-#> $ hireable : NULL
-#> $ bio : chr "Chief Scientist at @RStudio"
-#> $ public_repos : int 206
-#> $ public_gists : int 160
-#> $ followers : int 10515
-#> $ following : int 7
-#> $ created_at : chr "2008-04-01T14:47:36Z"
-#> $ updated_at : chr "2017-08-14T19:03:52Z"The API might return invalid data, but this should be rare, so you can just rely on the parser to provide a useful error message.
-Turn API errors into R errors
-Next, you need to make sure that your API wrapper throws an error if the request failed. Using a web API introduces additional possible points of failure into R code aside from those occurring in R itself. These include:
--
-
- Client-side exceptions -
- Network / communication exceptions -
- Server-side exceptions -
You need to make sure these are all converted into regular R errors. You can figure out if there’s a problem with http_error(), which checks the HTTP status code. Status codes in the 400 range usually mean that you’ve done something wrong. Status codes in the 500 range typically mean that something has gone wrong on the server side.
Often the API will provide information about the error in the body of the response: you should use this where available. If the API returns special errors for common problems, you might want to provide more detail in the error. For example, if you run out of requests and are rate limited you might want to tell the user how long to wait until they can make the next request (or even automatically wait that long!).
-github_api <- function(path) {
- url <- modify_url("https://api.github.com", path = path)
-
- resp <- GET(url)
- if (http_type(resp) != "application/json") {
- stop("API did not return json", call. = FALSE)
- }
-
- parsed <- jsonlite::fromJSON(content(resp, "text"), simplifyVector = FALSE)
-
- if (http_error(resp)) {
- stop(
- sprintf(
- "GitHub API request failed [%s]\n%s\n<%s>",
- status_code(resp),
- parsed$message,
- parsed$documentation_url
- ),
- call. = FALSE
- )
- }
-
- structure(
- list(
- content = parsed,
- path = path,
- response = resp
- ),
- class = "github_api"
- )
-}
-github_api("/user/hadley")
-#> Error: GitHub API request failed [404]
-#> Not Found
-#> <https://developer.github.com/v3>--Some poorly written APIs will return different types of response based on whether or not the request succeeded or failed. If your API does this you’ll need to make your request function check the
-status_code()before parsing the response.
For many APIs, the common approach is to retry API calls that return something in the 500 range. However, when doing this, it’s extremely important to make sure to do this with some form of exponential backoff: if something’s wrong on the server-side, hammering the server with retries may make things worse, and may lead to you exhausting quota (or hitting other sorts of rate limits). A common policy is to retry up to 5 times, starting at 1s, and each time doubling and adding a small amount of jitter (plus or minus up to, say, 5% of the current wait time).
-Set a user agent
-While we’re in this function, there’s one important header that you should set for every API wrapper: the user agent. The user agent is a string used to identify the client. This is most useful for the API owner as it allows them to see who is using the API. It’s also useful for you if you have a contact on the inside as it often makes it easier for them to pull your requests from their logs and see what’s going wrong. If you’re hitting a commercial API, this also makes it easier for internal R advocates to see how many people are using their API via R and hopefully assign more resources.
-A good default for an R API package wrapper is to make it the URL to your GitHub repo:
-ua <- user_agent("http://github.com/hadley/httr")
-ua
-#> <request>
-#> Options:
-#> * useragent: http://github.com/hadley/httr
-
-github_api <- function(path) {
- url <- modify_url("https://api.github.com", path = path)
-
- resp <- GET(url, ua)
- if (http_type(resp) != "application/json") {
- stop("API did not return json", call. = FALSE)
- }
-
- parsed <- jsonlite::fromJSON(content(resp, "text"), simplifyVector = FALSE)
-
- if (status_code(resp) != 200) {
- stop(
- sprintf(
- "GitHub API request failed [%s]\n%s\n<%s>",
- status_code(resp),
- parsed$message,
- parsed$documentation_url
- ),
- call. = FALSE
- )
- }
-
- structure(
- list(
- content = parsed,
- path = path,
- response = resp
- ),
- class = "github_api"
- )
-}Passing parameters
-Most APIs work by executing an HTTP method on a specified URL with some additional parameters. These parameters can be specified in a number of ways, including in the URL path, in URL query arguments, in HTTP headers, and in the request body itself. These parameters can be controlled using httr functions:
--
-
- URL path:
modify_url()
- - Query arguments: The
queryargument toGET(),POST(), etc.
- - HTTP headers:
add_headers()
- - Request body: The
bodyargument toGET(),POST(), etc.
-
RESTful APIs also use the HTTP verb to communicate arguments (e.g., GET retrieves a file, POST adds a file, DELETE removes a file, etc.). We can use the helpful httpbin service to show how to send arguments in each of these ways.
# modify_url
-POST(modify_url("https://httpbin.org", path = "/post"))
-
-# query arguments
-POST("http://httpbin.org/post", query = list(foo = "bar"))
-
-# headers
-POST("http://httpbin.org/post", add_headers(foo = "bar"))
-
-# body
-## as form
-POST("http://httpbin.org/post", body = list(foo = "bar"), encode = "form")
-## as json
-POST("http://httpbin.org/post", body = list(foo = "bar"), encode = "json")Many APIs will use just one of these forms of argument passing, but others will use multiple of them in combination. Best practice is to insulate the user from how and where the various arguments are used by the API and instead simply expose relevant arguments via R function arguments, some of which might be used in the URL, in the headers, in the body, etc.
-If a parameter has a small fixed set of possible values that are allowed by the API, you can use list them in the default arguments and then use match.arg() to ensure that the caller only supplies one of those values. (This also allows the user to supply the short unique prefixes.)
f <- function(x = c("apple", "banana", "orange")) {
- match.arg(x)
-}
-f("a")
-#> [1] "apple"It is good practice to explicitly set default values for arguments that are not required to NULL. If there is a default value, it should be the first one listed in the vector of allowed arguments.
Authentication
-Many APIs can be called without any authentication (just as if you called them in a web browser). However, others require authentication to perform particular requests or to avoid rate limits and other limitations. The most common forms of authentication are OAuth and HTTP basic authentication:
--
-
“Basic” authentication: This requires a username and password (or sometimes just a username). This is passed as part of the HTTP request. In httr, you can do:
GET("http://httpbin.org", authenticate("username", "password"))
-Basic authentication with an API key: An alternative provided by many APIs is an API “key” or “token” which is passed as part of the request. It is better than a username/password combination because it can be regenerated independent of the username and password.
-This API key can be specified in a number of different ways: in a URL query argument, in an HTTP header such as the
Authorizationheader, or in an argument inside the request body.
-OAuth: OAuth is a protocol for generating a user- or session-specific authentication token to use in subsequent requests. (An early standard, OAuth 1.0, is not terribly common any more. See
oauth1.0_token()for details.) The current OAuth 2.0 standard is very common in modern web apps. It involves a round trip between the client and server to establish if the API client has the authority to access the data. Seeoauth2.0_token(). It’s ok to publish the app ID and app “secret” - these are not actually important for security of user data.
-
--Some APIs describe their authentication processes inaccurately, so care needs to be taken to understand the true authentication mechanism regardless of the label used in the API docs.
-
It is possible to specify the key(s) or token(s) required for basic or OAuth authentication in a number of different ways (see Appendix for a detailed discussion). You may also need some way to preserve user credentials between function calls so that end users do not need to specify them each time. A good start is to use an environment variable. Here is an example of how to write a function that checks for the presence of a GitHub personal access token and errors otherwise:
-github_pat <- function() {
- pat <- Sys.getenv('GITHUB_PAT')
- if (identical(pat, "")) {
- stop("Please set env var GITHUB_PAT to your github personal access token",
- call. = FALSE)
- }
-
- pat
-}Pagination (handling multi-page responses)
-One particularly frustrating aspect of many APIs is dealing with paginated responses. This is common in APIs that offer search functionality and have the potential to return a very large number of responses. Responses might be paginated because there is a large number of response elements or because elements are updated frequently. Often they will be sorted by an explicit or implicit argument specified in the request.
-When a response is paginated, the API response will typically respond with a header or value specified in the body that contains one of the following:
--
-
- The total number of pages of responses -
- The total number of response elements (with multiple elements per page) -
- An indicator for whether any further elements or pages are available. -
- A URL containing the next page -
These values can then be used to make further requests. This will either involve specifying a specific page of responses or specifying a “next page token” that returns the next page of results. How to deal with pagination is a difficult question and a client could implement any of the following:
--
-
- Return one page only by default with an option to return additional specific pages -
- Return a specified page (defaulting to 1) and require the end user to handle pagination -
- Return all pages by writing an internal process of checking for further pages and combining the results -
The choice of which to use depends on your needs and goals and the rate limits of the API.
-Rate limiting
-Many APIs are rate limited, which means that you can only send a certain number of requests per hour. Often if your request is rate limited, the error message will tell you how long you should wait before performing another request. You might want to expose this to the user, or even include a wall to Sys.sleep() that waits long enough.
For example, we could implement a rate_limit() function that tells you how many calls against the github API are available to you.
rate_limit <- function() {
- github_api("/rate_limit")
-}
-rate_limit()
-#> <GitHub /rate_limit>
-#> List of 2
-#> $ resources:List of 3
-#> ..$ core :List of 3
-#> .. ..$ limit : int 60
-#> .. ..$ remaining: int 56
-#> .. ..$ reset : int 1503082049
-#> ..$ search :List of 3
-#> .. ..$ limit : int 10
-#> .. ..$ remaining: int 10
-#> .. ..$ reset : int 1503078516
-#> ..$ graphql:List of 3
-#> .. ..$ limit : int 0
-#> .. ..$ remaining: int 0
-#> .. ..$ reset : int 1503082056
-#> $ rate :List of 3
-#> ..$ limit : int 60
-#> ..$ remaining: int 56
-#> ..$ reset : int 1503082049After getting the first version working, you’ll often want to polish the output to be more user friendly. For this example, we can parse the unix timestamps into more useful date types.
-rate_limit <- function() {
- req <- github_api("/rate_limit")
- core <- req$content$resources$core
-
- reset <- as.POSIXct(core$reset, origin = "1970-01-01")
- cat(core$remaining, " / ", core$limit,
- " (Resets at ", strftime(reset, "%H:%M:%S"), ")\n", sep = "")
-}
-
-rate_limit()
-#> 56 / 60 (Resets at 13:47:29)Getting started with httr
- - - -httr quickstart guide
-The goal of this document is to get you up and running with httr as quickly as possible. httr is designed to map closely to the underlying http protocol. I’ll try and explain the basics in this intro, but I’d also recommend “HTTP: The Protocol Every Web Developer Must Know” or “HTTP made really easy”.
-This vignette (and parts of the httr API) derived from the excellent “Requests quickstart guide” by Kenneth Reitz. Requests is a python library similar in spirit to httr.
-There are two important parts to http: the request, the data sent to the server, and the response, the data sent back from the server. In the first section, you’ll learn about the basics of constructing a request and accessing the response. In the second and third sections, you’ll dive into more details of each.
-httr basics
-To make a request, first load httr, then call GET() with a url:
library(httr)
-r <- GET("http://httpbin.org/get")This gives you a response object. Printing a response object gives you some useful information: the actual url used (after any redirects), the http status, the file (content) type, the size, and if it’s a text file, the first few lines of output.
-r
-#> Response [http://httpbin.org/get]
-#> Date: 2017-08-18 17:47
-#> Status: 200
-#> Content-Type: application/json
-#> Size: 329 B
-#> {
-#> "args": {},
-#> "headers": {
-#> "Accept": "application/json, text/xml, application/xml, */*",
-#> "Accept-Encoding": "gzip, deflate",
-#> "Connection": "close",
-#> "Host": "httpbin.org",
-#> "User-Agent": "libcurl/7.54.0 r-curl/2.8.1 httr/1.3.1"
-#> },
-#> "origin": "104.153.224.166",
-#> ...You can pull out important parts of the response with various helper methods, or dig directly into the object:
-status_code(r)
-#> [1] 200
-headers(r)
-#> $connection
-#> [1] "keep-alive"
-#>
-#> $server
-#> [1] "meinheld/0.6.1"
-#>
-#> $date
-#> [1] "Fri, 18 Aug 2017 17:47:39 GMT"
-#>
-#> $`content-type`
-#> [1] "application/json"
-#>
-#> $`access-control-allow-origin`
-#> [1] "*"
-#>
-#> $`access-control-allow-credentials`
-#> [1] "true"
-#>
-#> $`x-powered-by`
-#> [1] "Flask"
-#>
-#> $`x-processed-time`
-#> [1] "0.00117087364197"
-#>
-#> $`content-length`
-#> [1] "329"
-#>
-#> $via
-#> [1] "1.1 vegur"
-#>
-#> attr(,"class")
-#> [1] "insensitive" "list"
-str(content(r))
-#> List of 4
-#> $ args : Named list()
-#> $ headers:List of 5
-#> ..$ Accept : chr "application/json, text/xml, application/xml, */*"
-#> ..$ Accept-Encoding: chr "gzip, deflate"
-#> ..$ Connection : chr "close"
-#> ..$ Host : chr "httpbin.org"
-#> ..$ User-Agent : chr "libcurl/7.54.0 r-curl/2.8.1 httr/1.3.1"
-#> $ origin : chr "104.153.224.166"
-#> $ url : chr "http://httpbin.org/get"I’ll use httpbin.org throughout this introduction. It accepts many types of http request and returns json that describes the data that it received. This makes it easy to see what httr is doing.
As well as GET(), you can also use the HEAD(), POST(), PATCH(), PUT() and DELETE() verbs. You’re probably most familiar with GET() and POST(): GET() is used by your browser when requesting a page, and POST() is (usually) used when submitting a form to a server. PUT(), PATCH() and DELETE() are used most often by web APIs.
The response
-The data sent back from the server consists of three parts: the status line, the headers and the body. The most important part of the status line is the http status code: it tells you whether or not the request was successful. I’ll show you how to access that data, then how to access the body and headers.
-The status code
-The status code is a three digit number that summarises whether or not the request was successful (as defined by the server that you’re talking to). You can access the status code along with a descriptive message using http_status():
r <- GET("http://httpbin.org/get")
-# Get an informative description:
-http_status(r)
-#> $category
-#> [1] "Success"
-#>
-#> $reason
-#> [1] "OK"
-#>
-#> $message
-#> [1] "Success: (200) OK"
-
-# Or just access the raw code:
-r$status_code
-#> [1] 200A successful request always returns a status of 200. Common errors are 404 (file not found) and 403 (permission denied). If you’re talking to web APIs you might also see 500, which is a generic failure code (and thus not very helpful). If you’d like to learn more, the most memorable guides are the http status cats.
-You can automatically throw a warning or raise an error if a request did not succeed:
-warn_for_status(r)
-stop_for_status(r)I highly recommend using one of these functions whenever you’re using httr inside a function (i.e. not interactively) to make sure you find out about errors as soon as possible.
-The body
-There are three ways to access the body of the request, all using content():
-
-
-content(r, "text")accesses the body as a character vector:-r <- GET("http://httpbin.org/get") -content(r, "text") -#> No encoding supplied: defaulting to UTF-8. -#> [1] "{\n \"args\": {}, \n \"headers\": {\n \"Accept\": \"application/json, text/xml, application/xml, */*\", \n \"Accept-Encoding\": \"gzip, deflate\", \n \"Connection\": \"close\", \n \"Host\": \"httpbin.org\", \n \"User-Agent\": \"libcurl/7.54.0 r-curl/2.8.1 httr/1.3.1\"\n }, \n \"origin\": \"104.153.224.166\", \n \"url\": \"http://httpbin.org/get\"\n}\n"httr will automatically decode content from the server using the encoding supplied in the
-content-typeHTTP header. Unfortunately you can’t always trust what the server tells you, so you can override encoding if needed:-content(r, "text", encoding = "ISO-8859-1")If you’re having problems figuring out what the correct encoding should be, try
stringi::stri_enc_detect(content(r, "raw")).
-For non-text requests, you can access the body of the request as a raw vector:
--content(r, "raw") -#> [1] 7b 0a 20 20 22 61 72 67 73 22 3a 20 7b 7d 2c 20 0a 20 20 22 68 65 61 -#> [24] 64 65 72 73 22 3a 20 7b 0a 20 20 20 20 22 41 63 63 65 70 74 22 3a 20 -#> [47] 22 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e 2c 20 74 65 78 74 -#> [70] 2f 78 6d 6c 2c 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 6d 6c 2c 20 -#> [93] 2a 2f 2a 22 2c 20 0a 20 20 20 20 22 41 63 63 65 70 74 2d 45 6e 63 6f -#> [116] 64 69 6e 67 22 3a 20 22 67 7a 69 70 2c 20 64 65 66 6c 61 74 65 22 2c -#> [139] 20 0a 20 20 20 20 22 43 6f 6e 6e 65 63 74 69 6f 6e 22 3a 20 22 63 6c -#> [162] 6f 73 65 22 2c 20 0a 20 20 20 20 22 48 6f 73 74 22 3a 20 22 68 74 74 -#> [185] 70 62 69 6e 2e 6f 72 67 22 2c 20 0a 20 20 20 20 22 55 73 65 72 2d 41 -#> [208] 67 65 6e 74 22 3a 20 22 6c 69 62 63 75 72 6c 2f 37 2e 35 34 2e 30 20 -#> [231] 72 2d 63 75 72 6c 2f 32 2e 38 2e 31 20 68 74 74 72 2f 31 2e 33 2e 31 -#> [254] 22 0a 20 20 7d 2c 20 0a 20 20 22 6f 72 69 67 69 6e 22 3a 20 22 31 30 -#> [277] 34 2e 31 35 33 2e 32 32 34 2e 31 36 36 22 2c 20 0a 20 20 22 75 72 6c -#> [300] 22 3a 20 22 68 74 74 70 3a 2f 2f 68 74 74 70 62 69 6e 2e 6f 72 67 2f -#> [323] 67 65 74 22 0a 7d 0aThis is exactly the sequence of bytes that the web server sent, so this is the highest fidelity way of saving files to disk:
-bin <- content(r, "raw") -writeBin(bin, "myfile.txt")
-httr provides a number of default parsers for common file types:
--# JSON automatically parsed into named list -str(content(r, "parsed")) -#> List of 4 -#> $ args : Named list() -#> $ headers:List of 5 -#> ..$ Accept : chr "application/json, text/xml, application/xml, */*" -#> ..$ Accept-Encoding: chr "gzip, deflate" -#> ..$ Connection : chr "close" -#> ..$ Host : chr "httpbin.org" -#> ..$ User-Agent : chr "libcurl/7.54.0 r-curl/2.8.1 httr/1.3.1" -#> $ origin : chr "104.153.224.166" -#> $ url : chr "http://httpbin.org/get"See
-?contentfor a complete list.These are convenient for interactive usage, but if you’re writing an API wrapper, it’s best to parse the text or raw content yourself and check it is as you expect. See the API wrappers vignette for more details.
-
The headers
-Access response headers with headers():
headers(r)
-#> $connection
-#> [1] "keep-alive"
-#>
-#> $server
-#> [1] "meinheld/0.6.1"
-#>
-#> $date
-#> [1] "Fri, 18 Aug 2017 17:47:42 GMT"
-#>
-#> $`content-type`
-#> [1] "application/json"
-#>
-#> $`access-control-allow-origin`
-#> [1] "*"
-#>
-#> $`access-control-allow-credentials`
-#> [1] "true"
-#>
-#> $`x-powered-by`
-#> [1] "Flask"
-#>
-#> $`x-processed-time`
-#> [1] "0.00101518630981"
-#>
-#> $`content-length`
-#> [1] "329"
-#>
-#> $via
-#> [1] "1.1 vegur"
-#>
-#> attr(,"class")
-#> [1] "insensitive" "list"This is basically a named list, but because http headers are case insensitive, indexing this object ignores case:
-headers(r)$date
-#> [1] "Fri, 18 Aug 2017 17:47:42 GMT"
-headers(r)$DATE
-#> [1] "Fri, 18 Aug 2017 17:47:42 GMT"The request
-Like the response, the request consists of three pieces: a status line, headers and a body. The status line defines the http method (GET, POST, DELETE, etc) and the url. You can send additional data to the server in the url (with the query string), in the headers (including cookies) and in the body of POST(), PUT() and PATCH() requests.
The url query string
-A common way of sending simple key-value pairs to the server is the query string: e.g. http://httpbin.org/get?key=val. httr allows you to provide these arguments as a named list with the query argument. For example, if you wanted to pass key1=value1 and key2=value2 to http://httpbin.org/get you could do:
r <- GET("http://httpbin.org/get",
- query = list(key1 = "value1", key2 = "value2")
-)
-content(r)$args
-#> $key1
-#> [1] "value1"
-#>
-#> $key2
-#> [1] "value2"Any NULL elements are automatically dropped from the list, and both keys and values are escaped automatically.
r <- GET("http://httpbin.org/get",
- query = list(key1 = "value 1", "key 2" = "value2", key2 = NULL))
-content(r)$args
-#> $`key 2`
-#> [1] "value2"
-#>
-#> $key1
-#> [1] "value 1"Custom headers
-You can add custom headers to a request with add_headers():
r <- GET("http://httpbin.org/get", add_headers(Name = "Hadley"))
-str(content(r)$headers)
-#> List of 7
-#> $ Accept : chr "application/json, text/xml, application/xml, */*"
-#> $ Accept-Encoding: chr "gzip, deflate"
-#> $ Connection : chr "close"
-#> $ Cookie : chr "a=1; b=1"
-#> $ Host : chr "httpbin.org"
-#> $ Name : chr "Hadley"
-#> $ User-Agent : chr "libcurl/7.54.0 r-curl/2.8.1 httr/1.3.1"(Note that content(r)$header retrieves the headers that httpbin received. headers(r) gives the headers that it sent back in its response.)
Fetching JSON data from REST APIs
-2017-05-31
- -This section lists some examples of public HTTP APIs that publish data in JSON format. These are great to get a sense of the complex structures that are encountered in real world JSON data. All services are free, but some require registration/authentication. Each example returns lots of data, therefore not all output is printed in this document.
-library(jsonlite)Github
-Github is an online code repository and has APIs to get live data on almost all activity. Below some examples from a well known R package and author:
-hadley_orgs <- fromJSON("https://api.github.com/users/hadley/orgs")
-hadley_repos <- fromJSON("https://api.github.com/users/hadley/repos")
-gg_commits <- fromJSON("https://api.github.com/repos/hadley/ggplot2/commits")
-gg_issues <- fromJSON("https://api.github.com/repos/hadley/ggplot2/issues")
-
-#latest issues
-paste(format(gg_issues$user$login), ":", gg_issues$title) [1] "jsta : fix broken stowers link"
- [2] "krlmlr : Log transform on geom_bar() silently omits layer"
- [3] "yutannihilation : Fix a broken link in README"
- [4] "raubreywhite : Fix theme_gray's legend/panels for large base_size"
- [5] "batuff : Add minor ticks to axes"
- [6] "mcol : overlapping boxes with geom_boxplot(varwidth=TRUE)"
- [7] "karawoo : Fix density calculations for groups with one or two elements"
- [8] "Thieffen : fix typo"
- [9] "Thieffen : fix typo"
-[10] "thjwong : `axis.line` works, but not `axis.line.x` and `axis.line.y`"
-[11] "schloerke : scale_discrete not listening to 'breaks' arg"
-[12] "hadley : Consider use of vwline"
-[13] "JTapper : geom_polygon accessing data$y"
-[14] "Ax3man : Added linejoin parameter to geom_segment."
-[15] "LSanselme : geom_density with groups of 1 or 2 elements"
-[16] "philstraforelli : (feature request) Changing facet_wrap strip colour based on variable in data frame"
-[17] "eliocamp : geom_tile() + coord_map() is extremely slow."
-[18] "eliocamp : facet_wrap() doesn't play well with expressions in facets. "
-[19] "dantonnoriega : Request: Quick visual example for each geom at http://ggplot2.tidyverse.org/reference/"
-[20] "randomgambit : it would be nice to have date_breaks('0.2 sec')"
-[21] "adrfantini : Labels can overlap in coord_sf()"
-[22] "adrfantini : borders() is incompatible with coord_sf() with projected coordinates"
-[23] "adrfantini : coord_proj() is superior to coord_map() and could be included in the default ggplot"
-[24] "adrfantini : Coordinates labels and gridlines are wrong in coord_map()"
-[25] "jonocarroll : Minor typo: monotonous -> monotonic"
-[26] "FabianRoger : label.size in geom_label is ignored when printing to pdf"
-[27] "andrewdolman : Add note recommending annotate"
-[28] "Henrik-P : scale_identity doesn't play well with guide = \"legend\""
-[29] "cpsievert : stat_sf(geom = \"text\")"
-[30] "hadley : Automatically fill in x for univariate boxplot" CitiBike NYC
-A single public API that shows location, status and current availability for all stations in the New York City bike sharing imitative.
-citibike <- fromJSON("http://citibikenyc.com/stations/json")
-stations <- citibike$stationBeanList
-colnames(stations) [1] "id" "stationName"
- [3] "availableDocks" "totalDocks"
- [5] "latitude" "longitude"
- [7] "statusValue" "statusKey"
- [9] "availableBikes" "stAddress1"
-[11] "stAddress2" "city"
-[13] "postalCode" "location"
-[15] "altitude" "testStation"
-[17] "lastCommunicationTime" "landMark" nrow(stations)[1] 666Ergast
-The Ergast Developer API is an experimental web service which provides a historical record of motor racing data for non-commercial purposes.
-res <- fromJSON('http://ergast.com/api/f1/2004/1/results.json')
-drivers <- res$MRData$RaceTable$Races$Results[[1]]$Driver
-colnames(drivers)[1] "driverId" "code" "url" "givenName"
-[5] "familyName" "dateOfBirth" "nationality" "permanentNumber"drivers[1:10, c("givenName", "familyName", "code", "nationality")] givenName familyName code nationality
-1 Michael Schumacher MSC German
-2 Rubens Barrichello BAR Brazilian
-3 Fernando Alonso ALO Spanish
-4 Ralf Schumacher SCH German
-5 Juan Pablo Montoya MON Colombian
-6 Jenson Button BUT British
-7 Jarno Trulli TRU Italian
-8 David Coulthard COU British
-9 Takuma Sato SAT Japanese
-10 Giancarlo Fisichella FIS ItalianProPublica
-Below an example from the ProPublica Nonprofit Explorer API where we retrieve the first 10 pages of tax-exempt organizations in the USA, ordered by revenue. The rbind_pages function is used to combine the pages into a single data frame.
#store all pages in a list first
-baseurl <- "https://projects.propublica.org/nonprofits/api/v1/search.json?order=revenue&sort_order=desc"
-pages <- list()
-for(i in 0:10){
- mydata <- fromJSON(paste0(baseurl, "&page=", i), flatten=TRUE)
- message("Retrieving page ", i)
- pages[[i+1]] <- mydata$filings
-}
-
-#combine all into one
-filings <- rbind_pages(pages)
-
-#check output
-nrow(filings)[1] 275filings[1:10, c("organization.sub_name", "organization.city", "totrevenue")] organization.sub_name organization.city totrevenue
-1 KAISER FOUNDATION HEALTH PLAN INC OAKLAND 40148558254
-2 KAISER FOUNDATION HEALTH PLAN INC OAKLAND 37786011714
-3 KAISER FOUNDATION HOSPITALS OAKLAND 20796549014
-4 KAISER FOUNDATION HOSPITALS OAKLAND 17980030355
-5 PARTNERS HEALTHCARE SYSTEM INC SOMERVILLE 10619215354
-6 UPMC PITTSBURGH 10098163008
-7 UAW RETIREE MEDICAL BENEFITS TR DETROIT 9890722789
-8 THRIVENT FINANCIAL FOR LUTHERANS MINNEAPOLIS 9475129863
-9 THRIVENT FINANCIAL FOR LUTHERANS MINNEAPOLIS 9021585970
-10 DIGNITY HEALTH SAN FRANCISCO 8718896265New York Times
-The New York Times has several APIs as part of the NYT developer network. These interface to data from various departments, such as news articles, book reviews, real estate, etc. Registration is required (but free) and a key can be obtained at here. The code below includes some example keys for illustration purposes.
-#search for articles
-article_key <- "&api-key=b75da00e12d54774a2d362adddcc9bef"
-url <- "http://api.nytimes.com/svc/search/v2/articlesearch.json?q=obamacare+socialism"
-req <- fromJSON(paste0(url, article_key))
-articles <- req$response$docs
-colnames(articles) [1] "web_url" "snippet" "lead_paragraph"
- [4] "abstract" "print_page" "blog"
- [7] "source" "multimedia" "headline"
-[10] "keywords" "pub_date" "document_type"
-[13] "news_desk" "section_name" "subsection_name"
-[16] "byline" "type_of_material" "_id"
-[19] "word_count" "slideshow_credits"#search for best sellers
-books_key <- "&api-key=76363c9e70bc401bac1e6ad88b13bd1d"
-url <- "http://api.nytimes.com/svc/books/v2/lists/overview.json?published_date=2013-01-01"
-req <- fromJSON(paste0(url, books_key))
-bestsellers <- req$results$list
-category1 <- bestsellers[[1, "books"]]
-subset(category1, select = c("author", "title", "publisher")) author title publisher
-1 Gillian Flynn GONE GIRL Crown Publishing
-2 John Grisham THE RACKETEER Knopf Doubleday Publishing
-3 E L James FIFTY SHADES OF GREY Knopf Doubleday Publishing
-4 Nicholas Sparks SAFE HAVEN Grand Central Publishing
-5 David Baldacci THE FORGOTTEN Grand Central Publishing#movie reviews
-movie_key <- "&api-key=b75da00e12d54774a2d362adddcc9bef"
-url <- "http://api.nytimes.com/svc/movies/v2/reviews/dvd-picks.json?order=by-date"
-req <- fromJSON(paste0(url, movie_key))
-reviews <- req$results
-colnames(reviews) [1] "display_title" "mpaa_rating" "critics_pick"
- [4] "byline" "headline" "summary_short"
- [7] "publication_date" "opening_date" "date_updated"
-[10] "link" "multimedia" reviews[1:5, c("display_title", "byline", "mpaa_rating")] display_title byline mpaa_rating
-1 Hermia & Helena GLENN KENNY
-2 The Women's Balcony NICOLE HERRINGTON
-3 Long Strange Trip DANIEL M. GOLD R
-4 Joshua: Teenager vs. Superpower KEN JAWOROWSKI
-5 Berlin Syndrome GLENN KENNY RSunlight Foundation
-The Sunlight Foundation is a non-profit that helps to make government transparent and accountable through data, tools, policy and journalism. Register a free key at here. An example key is provided.
-key <- "&apikey=39c83d5a4acc42be993ee637e2e4ba3d"
-
-#Find bills about drones
-drone_bills <- fromJSON(paste0("http://openstates.org/api/v1/bills/?q=drone", key))
-drone_bills$title <- substring(drone_bills$title, 1, 40)
-print(drone_bills[1:5, c("title", "state", "chamber", "type")]) title state chamber type
-1 AIRPORT AUTHORITIES-DRONES il upper bill
-2 Study Drone Use By Public Safety Agencie co lower bill
-3 AIRCRAFT/AVIATION: Provides for the exc la upper bill
-4 relative to the use of drones. nh lower bill
-5 Use or Operation of a Drone by Certain O fl lower bill#Local legislators
-legislators <- fromJSON(paste0("http://congress.api.sunlightfoundation.com/",
- "legislators/locate?latitude=42.96&longitude=-108.09", key))
-subset(legislators$results, select=c("last_name", "chamber", "term_start", "twitter_id")) last_name chamber term_start twitter_id
-1 Cheney house 2017-01-03 RepLizCheney
-2 Enzi senate 2015-01-06 SenatorEnzi
-3 Barrasso senate 2013-01-03 SenJohnBarrassoThe twitter API requires OAuth2 authentication. Some example code:
-#Create your own appication key at https://dev.twitter.com/apps
-consumer_key = "EZRy5JzOH2QQmVAe9B4j2w";
-consumer_secret = "OIDC4MdfZJ82nbwpZfoUO4WOLTYjoRhpHRAWj6JMec";
-
-#Use basic auth
-secret <- jsonlite::base64_enc(paste(consumer_key, consumer_secret, sep = ":"))
-req <- httr::POST("https://api.twitter.com/oauth2/token",
- httr::add_headers(
- "Authorization" = paste("Basic", gsub("\n", "", secret)),
- "Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"
- ),
- body = "grant_type=client_credentials"
-);
-
-#Extract the access token
-httr::stop_for_status(req, "authenticate with twitter")
-token <- paste("Bearer", httr::content(req)$access_token)
-
-#Actual API call
-url <- "https://api.twitter.com/1.1/statuses/user_timeline.json?count=10&screen_name=Rbloggers"
-req <- httr::GET(url, httr::add_headers(Authorization = token))
-json <- httr::content(req, as = "text")
-tweets <- fromJSON(json)
-substring(tweets$text, 1, 100) [1] "simmer 3.6.2 https://t.co/rRxgY2Ypfa #rstats #DataScience"
- [2] "Getting data for every Census tract in the US with purrr and tidycensus https://t.co/B3NYJS8sLO #rst"
- [3] "Gender Roles with Text Mining and N-grams https://t.co/Rwj0IaTiAR #rstats #DataScience"
- [4] "Data Science Podcasts https://t.co/SaAuO82a7M #rstats #DataScience"
- [5] "Reflections on ROpenSci Unconference 2017 https://t.co/87kMldvrsd #rstats #DataScience"
- [6] "Summarizing big data in R https://t.co/GMaZZ9sWiL #rstats #DataScience"
- [7] "Mining CRAN DESCRIPTION Files https://t.co/gWEIAYaBZF #rstats #DataScience"
- [8] "New package polypoly (helper functions for orthogonal polynomials) https://t.co/MzzzcIySym #rstats #"
- [9] "Hospital Infection Scores – R Shiny App https://t.co/Rf8wKNBPU6 #rstats #DataScience"
-[10] "New R job: Software Engineer in Test for RStudio https://t.co/X1bWkKlzYv #rstats #DataScience #jobs" Combining pages of JSON data with jsonlite
-2017-05-31
- -The jsonlite package is a JSON parser/generator for R which is optimized for pipelines and web APIs. It is used by the OpenCPU system and many other packages to get data in and out of R using the JSON format.
A bidirectional mapping
-One of the main strengths of jsonlite is that it implements a bidirectional mapping between JSON and data frames. Thereby it can convert nested collections of JSON records, as they often appear on the web, immediately into the appropriate R structure. For example to grab some data from ProPublica we can simply use:
library(jsonlite)
-mydata <- fromJSON("https://projects.propublica.org/forensics/geos.json", flatten = TRUE)
-View(mydata)The mydata object is a data frame which can be used directly for modeling or visualization, without the need for any further complicated data manipulation.
Paging with jsonlite
-A question that comes up frequently is how to combine pages of data. Most web APIs limit the amount of data that can be retrieved per request. If the client needs more data than what can fits in a single request, it needs to break down the data into multiple requests that each retrieve a fragment (page) of data, not unlike pages in a book. In practice this is often implemented using a page parameter in the API. Below an example from the ProPublica Nonprofit Explorer API where we retrieve the first 3 pages of tax-exempt organizations in the USA, ordered by revenue:
baseurl <- "https://projects.propublica.org/nonprofits/api/v1/search.json?order=revenue&sort_order=desc"
-mydata0 <- fromJSON(paste0(baseurl, "&page=0"), flatten = TRUE)
-mydata1 <- fromJSON(paste0(baseurl, "&page=1"), flatten = TRUE)
-mydata2 <- fromJSON(paste0(baseurl, "&page=2"), flatten = TRUE)
-
-#The actual data is in the filings element
-mydata0$filings[1:10, c("organization.sub_name", "organization.city", "totrevenue")] organization.sub_name organization.city totrevenue
-1 KAISER FOUNDATION HEALTH PLAN INC OAKLAND 40148558254
-2 KAISER FOUNDATION HEALTH PLAN INC OAKLAND 37786011714
-3 KAISER FOUNDATION HOSPITALS OAKLAND 20796549014
-4 KAISER FOUNDATION HOSPITALS OAKLAND 17980030355
-5 PARTNERS HEALTHCARE SYSTEM INC SOMERVILLE 10619215354
-6 UPMC PITTSBURGH 10098163008
-7 UAW RETIREE MEDICAL BENEFITS TR DETROIT 9890722789
-8 THRIVENT FINANCIAL FOR LUTHERANS MINNEAPOLIS 9475129863
-9 THRIVENT FINANCIAL FOR LUTHERANS MINNEAPOLIS 9021585970
-10 DIGNITY HEALTH SAN FRANCISCO 8718896265To analyze or visualize these data, we need to combine the pages into a single dataset. We can do this with the rbind_pages function. Note that in this example, the actual data is contained by the filings field:
#Rows per data frame
-nrow(mydata0$filings)[1] 25#Combine data frames
-filings <- rbind_pages(
- list(mydata0$filings, mydata1$filings, mydata2$filings)
-)
-
-#Total number of rows
-nrow(filings)[1] 75Automatically combining many pages
-We can write a simple loop that automatically downloads and combines many pages. For example to retrieve the first 20 pages with non-profits from the example above:
-#store all pages in a list first
-baseurl <- "https://projects.propublica.org/nonprofits/api/v1/search.json?order=revenue&sort_order=desc"
-pages <- list()
-for(i in 0:20){
- mydata <- fromJSON(paste0(baseurl, "&page=", i))
- message("Retrieving page ", i)
- pages[[i+1]] <- mydata$filings
-}
-
-#combine all into one
-filings <- rbind_pages(pages)
-
-#check output
-nrow(filings)[1] 525colnames(filings) [1] "tax_prd" "tax_prd_yr"
- [3] "formtype" "pdf_url"
- [5] "updated" "totrevenue"
- [7] "totfuncexpns" "totassetsend"
- [9] "totliabend" "pct_compnsatncurrofcr"
- [11] "tax_pd" "subseccd"
- [13] "unrelbusinccd" "initiationfees"
- [15] "grsrcptspublicuse" "grsincmembers"
- [17] "grsincother" "totcntrbgfts"
- [19] "totprgmrevnue" "invstmntinc"
- [21] "txexmptbndsproceeds" "royaltsinc"
- [23] "grsrntsreal" "grsrntsprsnl"
- [25] "rntlexpnsreal" "rntlexpnsprsnl"
- [27] "rntlincreal" "rntlincprsnl"
- [29] "netrntlinc" "grsalesecur"
- [31] "grsalesothr" "cstbasisecur"
- [33] "cstbasisothr" "gnlsecur"
- [35] "gnlsothr" "netgnls"
- [37] "grsincfndrsng" "lessdirfndrsng"
- [39] "netincfndrsng" "grsincgaming"
- [41] "lessdirgaming" "netincgaming"
- [43] "grsalesinvent" "lesscstofgoods"
- [45] "netincsales" "miscrevtot11e"
- [47] "compnsatncurrofcr" "othrsalwages"
- [49] "payrolltx" "profndraising"
- [51] "txexmptbndsend" "secrdmrtgsend"
- [53] "unsecurednotesend" "retainedearnend"
- [55] "totnetassetend" "nonpfrea"
- [57] "gftgrntsrcvd170" "txrevnuelevied170"
- [59] "srvcsval170" "grsinc170"
- [61] "grsrcptsrelated170" "totgftgrntrcvd509"
- [63] "grsrcptsadmissn509" "txrevnuelevied509"
- [65] "srvcsval509" "subtotsuppinc509"
- [67] "totsupp509" "ein"
- [69] "organization" "eostatus"
- [71] "tax_yr" "operatingcd"
- [73] "assetcdgen" "transinccd"
- [75] "subcd" "grscontrgifts"
- [77] "intrstrvnue" "dividndsamt"
- [79] "totexcapgn" "totexcapls"
- [81] "grsprofitbus" "otherincamt"
- [83] "compofficers" "contrpdpbks"
- [85] "totrcptperbks" "totexpnspbks"
- [87] "excessrcpts" "totexpnsexempt"
- [89] "netinvstinc" "totaxpyr"
- [91] "adjnetinc" "invstgovtoblig"
- [93] "invstcorpstk" "invstcorpbnd"
- [95] "totinvstsec" "fairmrktvalamt"
- [97] "undistribincyr" "cmpmininvstret"
- [99] "sec4940notxcd" "sec4940redtxcd"
-[101] "infleg" "contractncd"
-[103] "claimstatcd" "propexchcd"
-[105] "brwlndmnycd" "furngoodscd"
-[107] "paidcmpncd" "trnsothasstscd"
-[109] "agremkpaycd" "undistrinccd"
-[111] "dirindirintcd" "invstjexmptcd"
-[113] "propgndacd" "excesshldcd"
-[115] "grntindivcd" "nchrtygrntcd"
-[117] "nreligiouscd" "grsrents"
-[119] "costsold" "totrcptnetinc"
-[121] "trcptadjnetinc" "topradmnexpnsa"
-[123] "topradmnexpnsb" "topradmnexpnsd"
-[125] "totexpnsnetinc" "totexpnsadjnet"
-[127] "othrcashamt" "mrtgloans"
-[129] "othrinvstend" "fairmrktvaleoy"
-[131] "mrtgnotespay" "tfundnworth"
-[133] "invstexcisetx" "sect511tx"
-[135] "subtitleatx" "esttaxcr"
-[137] "txwithldsrc" "txpaidf2758"
-[139] "erronbkupwthld" "estpnlty"
-[141] "balduopt" "crelamt"
-[143] "tfairmrktunuse" "distribamt"
-[145] "adjnetinccola" "adjnetinccolb"
-[147] "adjnetinccolc" "adjnetinccold"
-[149] "adjnetinctot" "qlfydistriba"
-[151] "qlfydistribb" "qlfydistribc"
-[153] "qlfydistribd" "qlfydistribtot"
-[155] "valassetscola" "valassetscolb"
-[157] "valassetscolc" "valassetscold"
-[159] "valassetstot" "qlfyasseta"
-[161] "qlfyassetb" "qlfyassetc"
-[163] "qlfyassetd" "qlfyassettot"
-[165] "endwmntscola" "endwmntscolb"
-[167] "endwmntscolc" "endwmntscold"
-[169] "endwmntstot" "totsuprtcola"
-[171] "totsuprtcolb" "totsuprtcolc"
-[173] "totsuprtcold" "totsuprttot"
-[175] "pubsuprtcola" "pubsuprtcolb"
-[177] "pubsuprtcolc" "pubsuprtcold"
-[179] "pubsuprttot" "grsinvstinca"
-[181] "grsinvstincb" "grsinvstincc"
-[183] "grsinvstincd" "grsinvstinctot" From here, we can go straight to analyzing the filings data without any further tedious data manipulation.
-Cryptographic Hashing in R
-2017-09-06
- -The functions sha1, sha256, sha512, md4, md5 and ripemd160 bind to the respective digest functions in OpenSSL’s libcrypto. Both binary and string inputs are supported and the output type will match the input type.
md5("foo")[1] "acbd18db4cc2f85cedef654fccc4a4d8"md5(charToRaw("foo"))md5 ac:bd:18:db:4c:c2:f8:5c:ed:ef:65:4f:cc:c4:a4:d8 Functions are fully vectorized for the case of character vectors: a vector with n strings will return n hashes.
-# Vectorized for strings
-md5(c("foo", "bar", "baz"))[1] "acbd18db4cc2f85cedef654fccc4a4d8" "37b51d194a7513e45b56f6524f2d51f2"
-[3] "73feffa4b7f6bb68e44cf984c85f6e88"Besides character and raw vectors we can pass a connection object (e.g. a file, socket or url). In this case the function will stream-hash the binary contents of the connection.
-# Stream-hash a file
-myfile <- system.file("CITATION")
-md5(file(myfile))md5 16:a3:1a:bf:39:26:86:31:f2:0e:14:78:bf:64:d6:59 Same for URLs. The hash of the R-3.1.1-win.exe below should match the one in md5sum.txt
# Stream-hash from a network connection
-md5(url("http://cran.us.r-project.org/bin/windows/base/old/3.1.1/R-3.1.1-win.exe"))Compare to digest
-Similar functionality is also available in the digest package, but with a slightly different interface:
-# Compare to digest
-library(digest)
-Attaching package: 'digest'The following object is masked from 'package:openssl':
-
- sha1digest("foo", "md5", serialize = FALSE)[1] "acbd18db4cc2f85cedef654fccc4a4d8"# Other way around
-digest(cars, skip = 0)[1] "1952fbc796e1b9e8a634006f5c6e5770"md5(serialize(cars, NULL))md5 19:52:fb:c7:96:e1:b9:e8:a6:34:00:6f:5c:6e:57:70 Generating Secure Random Numbers in R
-2017-09-06
- -The rand_bytes function binds to RAND_bytes in OpenSSL to generate cryptographically strong pseudo-random bytes. See the OpenSSL documentation for what this means.
rnd <- rand_bytes(10)
-print(rnd) [1] 67 79 cc 0d a1 c7 10 4c d7 cbBytes are 8 bit and hence can have 2^8 = 256 possible values.
as.numeric(rnd) [1] 103 121 204 13 161 199 16 76 215 203Each random byte can be decomposed into 8 random bits (booleans)
-x <- rand_bytes(1)
-as.logical(rawToBits(x))[1] FALSE TRUE FALSE FALSE TRUE FALSE TRUE FALSESecure Random Numbers
-rand_num is a simple (2 lines) wrapper to rand_bytes to generate random numbers (doubles) between 0 and 1.
rand_num(10) [1] 0.1434106 0.2961425 0.3155963 0.9416567 0.5237373 0.6298659 0.5982151
- [8] 0.2486845 0.5146982 0.9181494To map random draws from [0,1] into a probability density, we can use a Cumulative Distribution Function. For example we can combine qnorm and rand_num to simulate rnorm:
# Secure rnorm
-x <- qnorm(rand_num(1000), mean = 100, sd = 15)
-hist(x)
Same for discrete distributions:
-# Secure rbinom
-y <- qbinom(rand_num(1000), size = 20, prob = 0.1)
-hist(y, breaks = -.5:(max(y)+1))