We are seeing a bunch of API failures when placing orders – please see a sample below – there’s no specific pattern observed since this is happening at irregular intervals and it’s happening on all actions - create / modify / delete. Could you help understand the cause of this?
Sometimes we also see HTTP return code 401 (expired access token) – however, there are successful placeorder API calls seen few milliseconds before and after the failure using the same token – couldn’t explain the cause – please advise.
2022-03-01T13:05:03.723-05:00 POST for order id RTYH21020D – 408 – Request Timeout – Request Timeout, error id: 26c01b3f-5e4e-44ab-9118-f8c2425d5351 2022/03/01/[$LATEST]ccfa7c06132d4da9ace9139b153978d9
2022-03-01T13:05:05.120-05:00 POST for order id YMH21032m30 – 408 – Request Timeout – Request Timeout, error id: 632adcf6-27df-4a9f-94f1-18c2d9e83363 2022/03/01/[$LATEST]ccfa7c06132d4da9ace9139b153978d9
2022-03-01T12:48:34.063-05:00 DELETE for order id RTYH21034m30 – 401 – Unauthorized – Access is denied 2022/03/01/[$LATEST]6d1581c6c7064056a2e0cc235965e73c
The 408 errors - apparently this means you’ve exceeded the maximum number of concurrent connections for your user. The max connections is 1 or 2 for a retail user - 1 if you are using a standard subscription, 2 if you pay for a subscription to Dual Connections (available in Application Settings → Subscriptions). I’d suggest looking at the execution of your restart process, see if maybe you try to log on while you’re already logged on. You can always call /auth/renewAccessToken to reset your token’s expiry.
At least the first 401 error you sent me ended up being an attempt to cancel an order that had been completed an archived the previous day. So in that case, look at how you get the data for a request to cancel an order. You can check an order’s execution status using it’s ordStatus field.
Thanks @Alexander , as always. I’ll review and be in touch.
Question for now regarding the max number of connections … If I’m logged in at https://trader.tradovate.com/ or the mobile app and then api, does that count as two connections or should the api connection still work? I see api orders still being placed when I’m logged in but of course we have the 408 errors every so often. I also see that I’m forcibly logged out from the web occasionally when several api orders are being placed together. I have the standard subscription.
To watch the Trader UI and use an app with your API key takes up both of your connections, if you have purchased Dual Subscriptions. Logging into Trader UI while running an app with only one max concurrent connection will result in the app’s connection being dropped and vice versa.
You could use just one app for both HTTP API and WebSocket API. They can be used interchangeably/situationally. But if you actually ran two apps with different API Keys, yes it would count as two connections.
Thanks @Alexander . That helps.
Just to confirm … Would using the same API key but two different tokens (one for REST API app and the other for Websocket app - I have different apps) count as two connections?
We have made changes so we are using only one token now but we still see 408s and 401s. We are using the one token between two apps (api and websocket).
Is there any point at which you attempt to get a new key or possibly store an old key? Is there any kind of consistency to the time that these errors occur?
Hi @fin1, chiming in to your query here. Do you know when you are getting the 401s? If it is happening around the 80 minute mark, it is definitely a renew access token issue.
After the token expires, it will no longer work for any REST orders until you renew it. If you received a successful response only milliseconds after getting the 401, I would assume that is just an asynchronous event that got back to you late.
We have been testing further and made some changes but we still have the same issue for some reason. Please see the explanation and follow-up questions below.
We have confirmed the app is using only one token and during this time the Trader App Web UI is also being used. We assume this means we are using 2 active connections. We thought we shouldn’t be seeing any more 401s and 408s since we have purchased two connections. However, we continue to see 401s and 408s just as before.
We are also using mdAccessToken for subscribing to market data (md/subscribeQuote) when ‘accessToken’ is being used for place order APIs and “user/syncrequest” websocket endpoint. Does this cause any issue?
Could you please advise possible next steps? In order to help you troubleshoot this further, the log events along with timestamps associated with this issue are provided below. We hope this helps to correlate events on your side.
Thanks!
Note : yPGTq********** is the active token at that time and it’s the only token active in the system used by all scaled instances (potentially multiple IP addresses in AWS where our app is deployed).
………….
[All times in ET time zone] 2022-03-13T20:00:09.648-04:00 getToken()::obtained token from auth api >> yPGTqWgWh1********
2022-03-13T20:00:10.473-04:00. !!!Retrying because of this response – Response:: PATCH for order id ZWK21003H4 – 401 – Unauthorized – Access is denied
Getting token again (Retry) 2022-03-13T20:00:10.482-04:00 getToken()::obtained token from auth api >> yPGTqWgWh1*************
(Retry failed) 2022-03-13T20:00:10.514-04:00 RAW response:: MODIFY/PATCH order for order id ZWK21003H4 – <Response [401]>
That mdAccessToken is really only there for backwards compatibility with the Trader application. For all regular API users, you should use accessToken. The mdAccessToken can be safely ignored. Looking into AWS IP handling may be worthwhile.
Also, I noticed you’re only documenting a 401 here. Are you still getting the 408s too, or just the 401s now?
We have more 408s than 401s. For example, there were 38 occurrences of 408 error and 1 occurrence of 401 between 3/13 9pm - 3/14 9am ET – last occurrence in this time span was at 2022-03-14T07:21:10.431-04:00 – all times in ET
We also had 21 occurrences of 1006 error during this time (screenshot attached).
We didn’t use the Trader app at all during this time so there were two connections for the api to use. At no point during this time, we were using more than one token for the api.
We have pretty much run out of ideas at this time. Appreciate if you could help troubleshoot/resolve this based on what you see at the Tradovate end. Thank you!
