Using Node's built-in HTTP/S modules

While working on a recent class project that involved MongoDB, Node and an external API, I tried creating HTTP request to an external API using just the built-in modules. I was thinking that this wouldn’t be too hard having done it before in Rails. As it turns out, I was going to have a bit of a harder time. I wasn’t getting my expected output: plain text. So, what was the problem?

First Problem

Instead of returning strings, the modules were returning streams. I’ll walk you through my thought process and, hopefully, you get something out of it.

My first instinct was to look for an HTTP module of some sort. I was thinking that this module would, hopefully, have some method implemented where I could create HTTP requests, not necessarily receive them. This didn’t present any problems because I was already using the HTTP module to create my server application. What I needed was to see the documentation for the module. I found it here.

The documentation told me that I needed an options object to be passed as a parameter to the http.get() function.

1
2
3
4

const reqObj = { protocol: 'http:',
  hostname: "api.hostname.com",
  path: "/path/to/use"
}

After doing so, I invoked the http.get() function. In the callback function, I tried printing out the response to the console.

1
2
3

http.get(options, function(res) {
  console.log(res);
});

This got gave me an error saying that the string “https” wasn’t a supported value for the protocol key. Digging a little deeper, I found that Node had an HTTPS module available. Using this, instead of the HTTP module, I came out with the following parameters:

1
2
3
4

const reqObj = {
  hostname: "api.hostname.com",
  path: "/path/to/use"
}

The output, this time, surprised me. It was a buffer with a bunch of HEX codes. It was in the format: <Buffer [hex code here] >. Upon seeing this output, I had to go back to the Node v6.x Documentation and check the type of the returned value. As it turns out, the https.get() function was returning a WritableStream through the clientRequest class. (This is also the case with the http.get() function). Looking at the sample code again, I noticed that it uses a toString() when writing out to console on the data event.

My code now looks like this:

1
2
3

https.get(options, function(res) {
  console.log(res.toString());
});

Side Note: In the HTTPS module documentation, they used toString() so that you could access the data. Meanwhile, in the HTTP module, their data event handler used process.stdOut.write(). These two methods seem to be the way to access the data output.

Another Problem

Now that I had my expected type, I went on to continue building the app logic. But, once I tried parsing the string to JSON, the string seemed to not hold all of the data. I was only returning one chunk of the data. If you haven’t worked with streams before, you’d definitely be surprised by this.

I was thinking that since I was being sent a buffer, maybe I wasn’t getting everything. Playing around with the code, I was able to come up with this:

1
2
3
4
5
6
7
8
9
10
11
12

const buffer = ''
https.get(getUrl, function(res) {
  res.on('data', (d) => {
    buffer += d;
  });
  res.on('end', () => {
    console.log(buffer)
  })
  res.on('error', (err) => {
    console.log(err)
  });
})

What I am doing is appending the data to an accumulator. When Node emits the end event, I will do what I want with the data, ie. returning from a Promise.

In doing this mental exercise of using the built-in modules of Node, I learned two things. First, the HTTP and HTTPS modules return a Stream. Second, one has to make sure that they have retrieved everything before parsing the string.

If that post was too long, here are my takeaways:

1. There is an HTTPS module bundled with Node.
2. https.get() and http.get() both return a class using the Writable Stream interface.
3. If you know that you are only supposed to have a very short response body for your GET call, you can use toString() or process.stdOut.write() to get to the data.
4. For longer responses, you might want to use the data event.