COSC203 Web, Databases, and Networks
Toggle Dark/Light/Auto modeToggle Dark/Light/Auto modeToggle Dark/Light/Auto modeBack to homepage

Lab 5: JavaScript, Fetching and Processing JSON

🎯 Lab Objective

In this lab, you will learn how to fetch JSON from a web server and process it.

This lab teaches the following:

  1. fetch()
  2. Processing JSON data
  3. GET and POST

Table of Contents

1. Create a new project

For the first part of this lab you should use a new project.

Create a new folder named lab05-fetch and create the files in that folder.

2. Fetching JSON

We are going to use the asynchronous data fetching function fetch() to get JSON data from a web server.

Fetching is not instant, servers take time to respond. This is why processing is handled with callback functions — the callback function will be called once the response arrives. Even after the response arrives, the associated data might not be finished downloading, so there is often another callback function for once download has been completed.

function responseCallback( response ) { ... }
function dataCallback( data ) { ... }

fetch("www.example.com").then( responseCallback ).then( dataCallback );
βœ… Tip

Reminder from the last lab — when passing a callback to a function only use the function’s name (i.e. without the parentheses)

foo( callback ); // correct βœ…
foo( callback() ); // incorrect ❌

The data we are going to fetch is a JSON array of emoji. The following is the URL for the JSON data file:

https://cosc203.cspages.otago.ac.nz/labs/files/emoji.json
πŸ“ Task 1: Fetching JSON
  1. Test the above link with a web browser. You should see some information about a bunch of emojis appear in JSON format.

  2. Create a new .js file in your project. This file is going to be used to fetch the emoji data for the chat app, so name the file appropriately.

  1. In the JS file, declare the JSON URL as a constant:

    const URL = "https://cosc203.cspages.otago.ac.nz/labs/files/emoji.json";
    
    
    
  2. Add the following three functions to your JS file:

    
    // callback for when the fetch response arrives
    function responseCallback(response) {
    
    }
    
    // callback for when the data is ready to use
    function dataReadyCallback(data) {
    
    }
    
    // callback for when a fetch error occurs
    function fetchErrorCallback(error) {
    
    }
    
  3. Add the fetch call to retrieve the JSON. This goes at the bottom of the JS file, at the root level.

    fetch(URL)
        .then(responseCallback)
        .then(dataReadyCallback)
        .catch(fetchErrorCallback);
    
  4. The responseCallback processes the server response.

    • if response.status != 200 then return; since something has gone wrong. The catch block will most likely be triggered in this case, so you don’t need any error handling code here.
    • if the status code is 200 then return the JSON text using: return response.text();
  5. The fetchErrorCallback is used to notify the user when the fetch fails, so add the following to that function:

    alert("An error occured when fetching the emoji data.  Please try again later.");
    console.log(error);
    
  6. The dataCallback processes the actual data.

    • you should first parse the JSON into JavaScript objects using:
    let emojis = JSON.parse(data);
    

    This will produce an array containing an object for each emoji.

  7. Loop over the emojis array (see below if you don’t remember how to do a for-each loop in JavaScript) and print all the emojis using console.log.

  8. To test the code, you have two options:

    1. Run the code using Node.js in a terminal with: node yourFileName.js

      This assumes that you have Node.js installed (which is installed on the lab machines).

    2. Create a simple HTML file and link the JS file via the script tag. You can then load the HTML file in the browser using the Live Server and look in the browser dev tools console for the output.

βœ… Arrays in JavaScript
let array = [1, 2, 3, 4, 5];

Use square brackets [ ] to access an element

console.log(array[0]); // prints 1

The .length property

console.log(array.length); // prints 5

For loops

for (let i = 0; i < array.length; i++) {
    console.log(array[i]);
}

For each loops

for (let element of array) {
    console.log(element);
}
βœ… Visual Studio Code

Use Visual Studio Code’s built-in terminal


3. Processing JSON

Often, we convert JSON between a string format and object format. Strings are useful as they can be stored as files (serialized) and sent across the internet, or stored directly in the browser’s localStorage, but objects are easier to work with in code.

To convert a JavaScript object into a string, use JSON.stringify()

const obj = {
    "name": "grinning face",
    "emoji": "πŸ˜€"
};
const string = JSON.stringify(obj);

To perform the inverse: convert a string into a JavaScript object, use JSON.parse()

const string = '{ "name": "grinning face", "emoji": "πŸ˜€" }';
const obj = JSON.parse(string);

In case you are wondering what JSON stands for — JavaScript Object Notation. It is the object literal format of JavaScript for creating objects. So, you can write JSON directly in a JavaScript file to create an object:

const grin = { "name": "grinning face", "emoji": "πŸ˜€" }
console.log(grin.emoji);

So, why do we need to use JSON.parse() — why don’t we just assign the JSON directly to a variable? Security. We can’t trust the JSON coming from a remote location to not include malicious functions. If we assigned it directly to a variable then those malicious functions would be included in our system. The JSON.parse() function will strip out any functions to ensure that what we end up with is safe to use in our application.

Dot Notation

The data we fetched is an array, so we can treat it as such.

Each element of the array is a normal JavaScript object with two data fields:

{
    "name": "smiling face with sunglasses",
    "emoji": "😎"
}

We can access each field with usual dot . notation for accessing the contents of an object:

let element = emojis[70];

console.log( element.name );

console.log( element.emoji );
πŸ“ Task 2: JSON processing
Modify your emoji fetching program to only print the emoji with "smiling" in their name.
βœ… Tip

You can do partial string matching in JavaScript with the string includes() method to search for substrings:

if ( myString.includes("smiling") ) { ... }

To make the partial search case-insensitive, you can convert both strings to lowercase first:

if ( myString.toLowerCase().includes(searchString.toLowerCase()) ){ ... }

You may find this tip useful in the assignment 1.


4. Chat Application

We are going to improve the chat application from the previous lab by using fetch() to communicate with an Web Service API.

Rather than chatting with Polly we will use a web service that allows you to chat with other people (or yourself using multiple browser tabs).

πŸ“ Task 3: A Better Chat Application

The provided files are modified from the previous lab.

We will modify lab05.js to fetch messages from a server (via a Web Service API).

  1. Download or clone the lab files from the following repository:

  2. Choose the correct URL for the web service at the top of lab05.js. If you are in the labs, or otherwise using the University network then the first URL is fine. If you are working from home, or outside of the University then use the second URL.

  3. Test the client app first!

    • in Visual Studio Code start Live Server extension.
    • Open lab05.html
    • Type a user name
    • Type a message, and hit Send
  4. Read the code!

    • It is important that you understand these functions:
      • lab05.js > submitNewMessage()
        • new ChatMessage(userName, message, time, "message-self")
      • lab05-utils.js > _create()
      • lab05-utils.js > _newelement()
    • They are refactored versions of the code that you saw in the previous lab.
  5. Test the server first!

  6. Add code to lab05.js > loadAllMessages() to fetch /messages.json

    • Use the pattern:
      fetch(url)
          .then(responseCallback)
          .then(dataCallback)
          .catch(errorCallback);
      
      • The responseCallback function should return response.text();
      • The dataCallback function should:
        • Parse the JSON JSON.parse(data) and store the resulting array in a variable.
        • For each message object in the array:
          • call new ChatMessage(name, message, time, type)
            • name, message, and time, are all in the message object
            • type is a CSS class and should be:
              • "message-self" if userName === message.name
              • "message-other" otherwise.
    • Messages you send will scroll off the top of the page since they aren’t actually being sent to the server (yet).
  7. Prevent duplicate messages

    • If you did everything right the chat log will grow infinitely 😒
    • This is because loadAllMessages() is called every 3 seconds (using setTimeout())
    • Modify your dataCallback function to wipe all messages before loading new ones
      • There is a clearMessages() function in lab05-utils.js that will clear the old messages.
      • Call that function before creating the new messages.
  8. Your application should now refresh the chat log every 3 seconds

    • This isn’t the best way to do this.
    • Can you think of a better way?

5. Posting Data with Fetch

By default fetch() sends GET requests. In the next task we will send POST requests to post your messages to the chat web service.

To send a POST request with fetch() you need to set the right options in the payload (the second parameter of fetch).

const fetchPayload = {
    method: "POST",
    body: '{ "name": "data" }', /* stringified JSON */
    headers: { "Content-Type": "application/json" }
}

fetch(API_URL, fetchPayload)
    .then(postResponseCallback)
    .then(postDataCallback)
    .catch (postErrorCallback);

Payload parameters:

  • method is the HTTP request type (GET is default).
  • body is the payload, the data we are sending to the server
  • headers are connection info and metadata.
πŸ“ Task 4: Create POST requests

Let’s send your messages to the service.

Write your code in lab05.js > submitNewMessage()

  1. Prepare the data for transmission

    • Format the message data into an object that can be passed as message object
    • it must have 3 data fields ("name", "message", and "time")
      • for time use new Date().getTime()
    const messageObj = {
        "name": userName,
        "message": message,
        "time": time
    };
    
  2. Use fetch to send the POST request

    • The server’s endpoint for receiving POST requests is: /send-message
    • fetch options:
    const postPayload = {
        method: 'POST',
        body: messageJSON, // stringify messageObj before sending!
        headers: { 'Content-Type': 'application/json' }
    }
    
  3. Write 3 callback functions to handle the response, data, and error callbacks.

  4. Send the request using

    fetch(API_URL + "/send-message", postPayload)
        .then(postResponseCallback)
        .then(postDataCallback)
        .catch(postErrorCallback);
    
  5. In the postResponseCallback(response) function

    • console.log(response.status);:
      • 201 means that the message was successfully POSTed
      • 400 means that what you sent was not correct
    • return response.text();
  6. In the postDataCallback(data) function

    • console.log(data); for debugging
  7. In the postErrorCallback(error) function

    • console.log(error) for debugging
  8. Congratulations

    • you can now chat with everyone in the lab (assuming you are using the shared chat service)!

6. Optional Challenge

Improve our chat client so it doesn’t wipe the chat log every 3 seconds.

πŸ“ Task 5: Optional Challenge
  1. Modify loadAllMessages() to be more efficient.
Don’t wipe the entire chat history. Instead, only create new messages if they aren’t yet loaded.
Use message.time to determine if the message has already been displayed.

Store the most recent message.time in a global variable.

If a message’s time is less than our global variable it’s already on our page.

If a message’s time is larger than our global variable, it needs to be displayed, so create a new message.


7. Marking Off

This lab is worth marks. be sure to get signed off.