Nodejs Tick Stream Prototype

API Developers
1

git repo

I’m a bit new to javascript so feel free to critique the source code. The project gets a key with an https request, authorizes a websocket connection from the key, makes an interface that waits for websocket responses and events, makes an interface that parses the chart events for ticks and presents them in order, and then prints the ticks. Most of this is done asynchronously, so that a request can be made and a response can be waited for or the next tick can be waited for and dealt with when it arrives. It would be nice to see what projects other people are writing to compare with.

The api seems workable so far. A few kinks that I ran into:

  • The example curl command for authentication on the api page has some typos. The appVersion field doesn’t have a comma after it and the \ symbol doesn’t concatenate lines when inside a single quote string.
  • The things I tried involving closestTimestamp and asFarAsTimestamp just gave me one event with an end of history flag. I ended up just using asMuchAsElements: 1.
  • There only seemed to be about an hour of historical tick data.

It would be nice to know if I’m messing something up with any of these or if it’s just a limitation of the api. What’s there so far seems like enough to do the sort of automatic trading that I’m interested in. For example, making a moving average from the history of ticks and trading long above the average and short below the average. Setting up orders, indicators, and setting maximum account losses is what I intend to work on next. Also is there a way to embed source code in a post or upload a text file? That would be more convenient than a git repo.

1 Like
2

Hi @fennel, I’m happy to try and help you out.

This may have to do with the contract you’re looking at or the timestamp you’re using. The timestamp has to be within the contract’s expiration.

Yes, you can wrap the code with three backticks (```) on either side.

this is a code block

if you paste your code in between those backticks it should format it. It would be helpful to see your source code.

If you are intending to use JavaScript, you should look at the API tutorial that we have. You seem like you have a good idea of what you want to do - for the more advanced websocket tutorials check out this section.

Edit: I noticed you did post a bitbucket with your source code. From what I can see, you’re using the API correctly. I’ll need some time to look at it and try to reproduce your tick data for closestTimestamp and asFarAsTimestamp.

4

If you are going to try to reproduce, then here’s a smaller example. If you give a closestTimestamp and an asFarAsTimestamp then getChart only produces a single event with an eoh flag.
example output, id is a number:

{
  "s": 200,
  "i": 0
}
{
  "s": 200,
  "i": 1,
  "d": {
    "mode": "RealTime",
    "historicalId": id,
    "realtimeId": id
  }
}
{
  "e": "chart",
  "d": {
    "charts": [
      {
        "id": id,
        "eoh": true
      }
    ]
  }
}

example source code

import { readFile, writeFile } from 'fs/promises'
import { request } from 'https'
//npm install ws
import WebSocket from 'ws'

const mdWsUrl = 'wss://md.tradovateapi.com/v1/websocket'

export const key = JSON.parse(await readFile('key.json')).key

//every request gets a unique id
async function makeNextId() {
  let count = 0
  return async () => {
    let oldCount = count
    count += 1
    return oldCount
  }
}

const ws = new WebSocket(mdWsUrl)
const nextId = await makeNextId()
const authorizeId = await nextId()
let resolveOpen
const openPromise = new Promise((resolve, reject) => { resolveOpen = resolve })
let resolveAuthorize
const authorizePromise = new Promise((resolve, reject) => { resolveAuthorize = resolve })

ws.on('message', async (message) => {
  try {
    const type = message.slice(0, 1)
    message = message.slice(1, message.length)
    //open frame
    if (type === 'o') {
      resolveOpen()
    }
    //heartbeat frame
    else if (type === 'h') {
      ws.send('[]')
    }
    //array frame
    else if (type === 'a') {
      const messages = JSON.parse(message)
      messages.forEach((eventOrResponse) => {
        console.log(JSON.stringify(eventOrResponse, null, 2))
        if (eventOrResponse['i'] === authorizeId) {
          resolveAuthorize(authorizeId)
        }
      })
    }
    //close frame
    else if (type === 'c') {
    }
    else { throw ('unknown message type ' + type) }
  }
  catch (err) {
    ws.close()
    console.error(err)
    process.exitCode = 1
  }
})

let startDate = new Date()
let endDate = new Date(startDate)
startDate.setUTCMinutes(startDate.getUTCMinutes() - 300)
await openPromise
ws.send('authorize' + '\n' + authorizeId + '\n' + '' + '\n' + key)
await authorizePromise
ws.send('md/getChart' + '\n' + (await nextId()) + '\n' + '' + '\n' + JSON.stringify({
  symbol: 'NQM1',
  chartDescription: {
    underlyingType: 'Tick',
    elementSize: 1,
    elementSizeUnit: 'UnderlyingUnits',
    withHistogram: false
  },
  timeRange: { closestTimestamp: startDate, asFarAsTimestamp: endDate }}))
5

Hello again @fennel,

I think the issue is that asFarAsTimestamp should be startDate and closestTimestamp should be endDate.

6

Oh wow that was definitely it. Thanks for that. Although the data only seems to go back 4096 ticks. It would be nice to be able to pull historical tick data for the past few weeks or the past year for backtesting strategies.

7

Hi. I am having a similar problem with only being able to receive 4096 ticks when I make a data request to the Tradovate API for historical tick data.

My question is, does Tradovate limit the user to only 4096 ticks max, if we do not pay for the Extended Tick Data package or can I get more than 4096 data points with a specific api request (without having to pay for the extended data package)?

Please advise @fennel or @Alexander if you can guide in me in the right direction. Thank you.

9

It’s pretty funny that people are still reading this post.

I never could get more than 4096 ticks or bars. I’m guessing that it’s a design choice when downloading historical data. The idea being to use getChart, then read the charts with the same id up to the end of history, and then sort them. Chart events and bars within a chart event don’t seem to necessarily come in order which is why getting small pieces at a time makes sense. Each batch of bars can be before a certain time instead of each bar being ordered. When I want a lot of historical data I just make more historical getChart requests and append the data. For example, I use asMuchAsElements: 4096 and closestTimestamp: new Date() which downloads 4096 bars from the current time. Then I sort the result and grab the timestamp of the first bar. Since closestTimestamp gets some amount of data up to but not including the timestamp more data can get retrieved with another getChart using asMuchAsElements: 4096 and closestTimestamp: new Date(bars[0].timestamp). This can be done repeatedly to get as many bars as desired. Something similar can be done to get data going back to a date.

I’ve reworked a lot of the code in the past 4 months, so I’ll probably make another post with code including rudimentary indicators, multiple chart support, and automated order examples.

10

Thanks for the reply @fennel . So with this method that you have described, can you get more than 4096 ticks of data now (using multiple batches) or are you still limited to 4096 in total?

11

The method would produce any amount of ticks collected in batches of 4096 ticks at a time. This is the nodejs code that I use for downloading range bars over a date range or downloading a number of bars from a specific time. It wouldn’t be that hard to change it to do ticks instead of range bars although range bars can use a lot less bandwidth for strategies that trade on longer time frames.

function findEoh(charts) {
  for (let i = 0; i < charts.length; i += 1) {
    if (charts[i].eoh !== undefined) { return i }
  }
  return -1
}

/*
api.getChart(body) takes a body for the getChart endpoint and returns a realtimeId corresponding to the activated chart
api.cancelChart(realtimeId) cances the chart corrseponding to realtimeId
api.nextCharts(realtimeId) returns the next charts event with any chart that isn't realtimeId filtered out
symbol will be something like MNQZ1
range is the number of ticks in a range bar, so range: 4 would be 1 point range bars, range:100 would be 25 point range bars, etc
*/
export async function makeHistory(options) {
  const { api, symbol, range } = options
  //either takes a startDate and an endDate, takes an elementsBack, or takes an elementsBack and an endDate
  let { startDate, endDate, elementsBack } = options
  if (endDate === undefined) { endDate = new Date() }

  async function chartsUpToEoh(realtimeId) {
    let allCharts = []
    let eohIndex
    let charts = await api.nextCharts(realtimeId)
    while ((eohIndex = findEoh(charts)) === -1) {
      for (const chart of charts) { allCharts.push(chart) }
      charts = await api.nextCharts(realtimeId)
    }
    charts = charts.slice(0, eohIndex)
    //ignore charts.slice(eohIndex + 1, charts.length) because those charts would be after the eoh
    for (const chart of charts) { allCharts.push(chart) }
    return allCharts
  }

  //collect all of the history given by a getChart
  async function singleHistoryTimeRange(timeRange) {
    const realtimeId = await api.getChart({
      symbol: symbol,
      chartDescription: {
        underlyingType: 'Tick',
        //size in ticks of the range bars
        elementSize: range,
        elementSizeUnit: 'Range',
        withHistogram: false
      },
      timeRange: timeRange})
    const charts = await chartsUpToEoh(realtimeId)
    await api.cancelChart(realtimeId)
    return charts
  }

  //convert charts into bars
  function prepareCharts(charts) {
    const bars = charts.map((chart) => { return chart.bars }).flat()
    //sort the bars in ascending order of timestamp
    bars.sort((a, b) => { return (new Date(a.timestamp)).getTime() - (new Date(b.timestamp)).getTime() })
    return bars
  }

  //return the bars from startDate up to but not including endDate, limited to around 4096 bars
  async function singleHistoryDates(options) {
    const { startDate, endDate } = options
    return prepareCharts(await singleHistoryTimeRange({ asFarAsTimestamp: startDate, closestTimestamp: endDate }))
  }

  //keep asking for more bars until the whole date range is there
  async function historyDates(options) {
    const { startDate, endDate } = options
    let array = []
    let date = new Date(endDate)
    let single = await singleHistoryDates({ startDate: startDate, endDate: date })
    while (single.length !== 0) {
      array.push(single)
      date = new Date(single[0].timestamp)
      single = await singleHistoryDates({ startDate: startDate, endDate: date })
    }
    return array.reverse().flat()
  }

  //start at an endDate and keep going back until there's a positive number of bars
  async function historyDatesUntilPositive(endDate) {
    //starts by looking 16 * 2 time units back and goes back longer if there's no bars
    let timeBack = 16
    let bars
    do {
      timeBack *= 2
      let startDate = new Date(endDate)
      startDate.setUTCMinutes(endDate.getUTCMinutes() - timeBack)
      bars = await historyDates({ startDate: startDate, endDate: endDate })
    }
    while (bars.length === 0)
    return bars
  }

  //start at an endDate and go back a number of bars
  async function singleHistoryElementsBack(options) {
    const { elementsBack, endDate } = options
    return prepareCharts(await singleHistoryTimeRange({ elementsBack: elementsBack, closestTimestamp: endDate }))
  }

  //keep asking for more bars until there's elementsBack bars
  async function historyElementsBack(options) {
    let { elementsBack, endDate } = options
    endDate = new Date(endDate)
    let array = []
    //sum the number of bars currently in array and keep asking for more bars until there's the same amount or more bars than elementsBack
    while ((array.map((a) => { return a.length }).reduce((accum, next) => { return accum + next }, 0)) < elementsBack) {
      //const bars = await historyDatesUntilPositive(endDate)
      const bars = await singleHistoryElementsBack({ elementsBack: 4096, endDate: endDate })
      endDate = new Date(bars[0].timestamp)
      array.push(bars)
    }
    //the amount of bars in the array is greater than or equal to elementsBack so trim it down to elementsBack bars
    const bars = array.reverse().flat()
    return bars.slice(bars.length - elementsBack, bars.length)
  }

  let bars
  if (startDate !== undefined) { bars = await historyDates({ startDate: startDate, endDate: endDate }) }
  else if (elementsBack !== undefined) { bars = await historyElementsBack({ elementsBack: elementsBack, endDate: endDate }) }
  else {
    console.error('options', options, 'are not correct')
    process.exitCode = 1
    process.exit()
  }
  return bars
}
1 Like
12

Hello @Alexander and @fennel . I had one other question regarding Tick data that I was hoping one of you had an answer for: Please advise if you have a solution.

What response should we get from the server if we are fetching tick data and we reach the max allowed to receive with a basic live account (and have not ordered the Extended tick data subscription)?
Should we get an empty array or do we automatically get the p-time/penalty?

I am trying to avoid getting an penalty time.