Corrupted JSON in message data

API Developers Community Support
1

FYI Brian mentioned this is a known issue but posting here for awareness.

One thing I noticed is that sometimes the messages seem corrupted and can’t be parsed (invalid JSON).

image
image909×189 50.5 KB

[{“e”:“md”,“d”:{“quotes”:[{“id”:333776,“timestamp”:“2021-06-15T12:40:30.842Z”,“contractId”:333776,“entries”:{“Bid”:{“price”:1865.9,“size”:5},“TotalTradeVolume”:{“size”:4,“Offer”:{“price”:1866.2,“size”:2},“LowPrice”:{“price”:1860.1},“Trade”:{“price”:1864.9,“size”:1},“OpenInterest”:{“size”:978},“OpeningPrice”:{“price”:1866.4},“HighPrice”:{“price”:1867.2},“SettlementPrice”:{“price”:1864}}}]}}]

The JSON looks normal but when I paste into my editor, it looks like this:

image

I am not sure what these characters are but might be something to check into.

2

To follow up on this, I’m seeing occasional issues with my data parsing related to null values for DOM prices. Appears most often around market close. Hope a fix is still in the works. Here is an example:

a[{"e":"md","d":{"doms":[{"contractId":1097759,"timestamp":"2021-08-03T21:00:00.258Z","bids":[{"price":2.1191,"size":1},{"price":2.1165,"size":1},{"price":2.11,"size":1},{"price":null,"size":0},{"price":2.1078,"size":15},{"price":null,"size":0},{"price":2.1,"size":13},{"price":null,"size":0},{"price":2.098,"size":6},{"price":null,"size":0},{"price":2.09,"size":1},{"price":null,"size":0},{"price":null,"size":0},{"price":2.089,"size":18},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":2.05,"size":1}],"offers":[{"price":2.1268,"size":3},{"price":2.13,"size":1},{"price":2.14,"size":1},{"price":2.15,"size":1},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":null,"size":0},{"price":2.1961,"size":6}]}]}}]

Would definitely prefer to have these null items in the “bids” and “offers” arrays omitted on the server side so they’re sent, like below:

a[{"e":"md","d":{"doms":[{"contractId":1097759,"timestamp":"2021-08-03T21:00:00.258Z","bids":[{"price":2.1191,"size":1},{"price":2.1165,"size":1},{"price":2.11,"size":1},{"price":2.1078,"size":15},{"price":2.1,"size":13},{"price":2.098,"size":6},{"price":2.09,"size":1},{"price":2.089,"size":18},{"price":2.05,"size":1}],"offers":[{"price":2.1268,"size":3},{"price":2.13,"size":1},{"price":2.14,"size":1},{"price":2.15,"size":1},{"price":2.1961,"size":6}]}]}}]

3

@Brady, I’m not 100% certain these issues are related, but for now ensure that in your code you check for null before performing an operation on your DOM data. There are lot’s of ways to do so in JS. You could use a simple if statement, or even set a default value with the || operator.

const domData = JSON.parse(rawData.slice(1))
domData.forEach(data => {
  const { bids, offers } = data.d
  bids.forEach(bid => {
    //check if price exists (doesn't coerce to falsey, which catches null/undefined)
    if(bid.price) { /*...*/ }
    // or set a default
    const price = bid.price || 0
  })
})

As for the known corrupt DOM data - what I personally do to prevent this from being an issue in my code is wrap the part of the program that decodes the JSON in a try catch block something like this:

// declare websocket
const WS = new WebSocket(MD_URL)

// add an event listener for the 'message' event
WS.addEventListener('message', function(msg) {
  let data
  // try to parse the JSON here 
  try {
    data = JSON.parse(msg.data.slice(1))
  } // logging as necessary in the catch block...
  catch(err) {
    data = []
    console.log('failed to process message: ' + err)
  }
})

// then we can carry on with the program

This solution will simply omit the corrupt data. Hope someone can find this useful!

1 Like
4

@Alexander Yep, try…catch has been working well for me with JSON parsing. I hadn’t had any issues until I started logging to CSV files. I changed JSON parsing modules, and the new module converts values it can’t interpret to null. So the eventual exception was pushed down the chain.

In Python, a useful check before storing to dataframes is isinstance(var, int) or isinstance(var, float).
In Julia, I’ve been using the check isa(var, Number).

5

I think we can mark this issue as resolved, haven’t seen this in quite a while. Curious if anyone else on this thread has seen it?