What we are mainly talking about here is the implementation of programmed real-time code for medium and low-frequency trading strategies.
Strategy
The birth of a quantitative trading strategy is generally based on observing and understanding the market, generating a strategy idea, then designing the specific details of the strategy, and then back-testing multiple varieties of data to obtain and verify the effect of specific parameter values. If the effect is good and has universal applicability, then the next step is to test the real market with small funds. If the out-of-sample real offer is feasible, you can eventually increase funds and run it for a long time.
In order to prevent leverage, let me explain that of course there are other methods to generate strategies, such as data mining, digging, digging, digging factors. There are also automatic generation factors such as machine learning and reinforcement learning. However, this kind of black box method is difficult to keep running during the strategy retracement period, because you don’t know when and why these dug-out factors will fail. When doing quantitative trading, the most important thing is to insist on firm trading. If the market is not good, you can shrink your position, but you cannot shut down. No one can know when the big market will break out. If you miss it, your previous retracement will be in vain.
An analogy is that the design and backtesting of strategies is equivalent to the "temple calculation" mentioned in "The Art of War" by Sun Tzu. "The one who has no battle but the temple is the winner has to calculate more; the one who has not fought but the temple is the loser has to calculate." Less means more. More means victory, less means no victory, and what’s more, nothing counts!”
Therefore, the trading strategy must be well planned and carefully calculated from the initial design, otherwise you will actually lose before the actual offer is made. This step is the most difficult. Although the subsequent actual code is not easy, most of it is just tedious. If you spend more time and effort, you can always do it well. The key to a firm offer is risk control.
Trading strategies, or trading rules, allow us to "do the right thing", and the specific implementation of the actual offer is to "do the right thing". Many times, knowing the right direction is far more difficult than walking the distance. Just like the joke about repairing equipment, it charges a thousand yuan to draw a line. Drawing the line itself is only worth 1 buck, but knowing where to draw it costs 999.
Strategy + real offer together form a trading system. Regarding the design of a trading system, you can refer to the previous article: Seven elements of a complete trading system. The real offer here is actually the specific operations of the next 4 steps of the trading system, "Entering and Exiting Profit and Loss". The first three steps are basically determined when the strategy design is completed.
However, even if the strategy is good, if the actual implementation is not in place, the effect will be greatly reduced, and even the potential strategy that was originally good will be stopped. Developing a good strategy is not easy, so don’t fail at executing it. A good idea needs to be firmly implemented in the end.
question
1. Sudden market situation
This kind of strategy will only be encountered if there is a signal within the bar. Within bar, the closing price or opening price of a certain K-line is not used, but the intraday price is used to trigger the signal.
Here is an example of a problem encountered in the real offer a few days ago. A microscopic example of a trading market. It is quite common in the currency circle.
The picture below is the 10-second K-line of the BNB perpetual contract a few days ago (2023.7.10 17:21) (a bar means OHLCV aggregated from transaction data within 10 seconds).
As you can see, the amplitude of the big positive line is 3.53%, and the transaction was about 30 million dollars. Completed in 10 seconds. Before this market started, the trend was almost uneventful.
Look at the picture below which is the 1 second k-line at that time, you can get more details.
It basically takes about 4 seconds to fully rise. It rose by more than 2 points in the first second. These are basically the results of high-frequency strategies (event-driven, high-frequency trends, market making, etc.) and algorithmic orders that may be arranged in advance.
Let’s look at BNB’s 4-hour K-line. The largest positive line is the time period when the news came out. Amplitude 4.05%.
Comparing the K-lines of these periods, we can see that this kind of news-driven fluctuations (this time the news is Binance’s new IEO, ARKM), are generally completed instantly. The increase of 4 hours is actually not much different from 10 seconds, and the real start-up is only about 4 seconds. This kind of battlefield belongs to those high-frequency strategies that are armed to the teeth. However, it may also affect mid- and low-frequency strategies, causing greater slippage.
We won’t study high-frequency strategies here. What does it mean for mid- and low-frequency strategies? There are two main things that come to mind:
1. Some trends break out very suddenly and are completed within a short period of time, so the strategy cannot be suspended casually. The real offer in the currency circle must be guaranteed to be online 24 hours a day, especially if there is a position, and it cannot be disconnected.
2. The actual slippage may reach 2 percentage points. Look at the 1-second K-line above. If your buy signal is based on a certain price, and this price happens to come out at the lower part of the big K-line, then no matter what, the price after you enter the market may slide out. 2~3 percentage points. The worst can be as low as 4 percentage points, and those buy orders are filled at the tip of the needle. However, if you are selling, you will make profits beyond your expectations. However, according to Murphy's law, the falling bread always lands buttered side. In the long run, there is a high probability that there will be more slippage.
The good thing, though, is that this happens only a limited number of times. As long as the strategy is fine and the slippage is larger, it is just profit taking and will not affect long-term positive returns. It's like enduring one more shock and wear.
One way to mitigate this is to use websocket to obtain the latest price. The document says it updates every 250 milliseconds, but sometimes it actually updates faster, so it is possible to respond immediately. Another advantage of websocket is that it does not occupy the frequency limit of the exchange's Restful API. This is mentioned in detail later.
2. The hourly quotation
This is encountered by many trend strategies, because most of them use the switch position price to calculate the entry and exit signals. Then as soon as the hour passes, everyone starts up one after another.
On the hour, especially multiples of 8, because no matter what cycle your strategy is, these three points should all meet the common divisor time. Maybe everyone’s automated strategies will take action, and the high, middle and low frequencies will be squeezed together to generate their own signals. , each other's rivals.
In addition, at this time, the exchange also needs settlement and delivery. Although the perpetual contract is not delivered, there may be other operations to calculate the delivery capital fee. For about 6 seconds after the three hours of 0/8/16, Binance’s websocket did not push K-line information, it stopped! Just ask you what to do? In reality, there is often no solution.
What's more serious is that the market will resonate, especially when the market is falling, and panic will be more contagious. The servers of the exchange are even busier, and many small coins with fierce market conditions are down immediately. Orders cannot be placed, and all are 1001 errors. When you place an order in a hurry and succeed, the orders traded on the tip of the pin may be yours.
Therefore, slippage is really unavoidable. It is fate. Just accept it. Over-optimizing the code will not be of much use. As mentioned before, just treat it as one more stop loss.
However, there are also some tips to share.
The signal runout can be calculated in advance n seconds before the hour. At that time, the congestion has not started, because most automated strategies may wait until the K-line closing price on the hour comes out before starting to calculate the signal. However, the problem caused by this operation is that it may generate false signals. For example, the price goes back immediately. There should not be a signal, and a false positive occurs. At this point it depends on your choice whether you want to do this or not. However, you can add a certain threshold and trigger it when it exceeds it, so as to prevent false signals caused by short-term price returns. The slippage can be reduced a little bit.
There is also a method of using offset signals such as time offset or shift k-line, which does not generate signals on the hour, does not get involved with others, and better avoids crowding. There are also methods such as TWAP.
However, these require some skills and will make the actual code more difficult.
firm offer
A basic mid- to low-frequency CTA strategy. Generally speaking, the actual code is very simple. There is only one strategy for a variety, and there is no addition or subtraction of positions, and no cross information is mixed together. Then the local code does not need to maintain any state, because the exchange records everything for you. Whether there is a position, how much margin is required, whether the order is completed, etc. Because it is medium to low frequency, you can check the exchange's account information at any time when needed, and this is the most immediate and accurate. This eliminates the trouble of using a local database to record transaction status.
This simple strategy is actually feasible, but the problem is that the retracement may be larger, not diversified enough, and the risk is high. A single product may have no market, fluctuate and retrace without turning back, or encounter extreme market conditions, and another big hole will appear on the capital curve. If you take it seriously, you will be frightened and stop your strategy if you cannot bear it. If you take it lightly, the trading experience will be much worse. Note that these are all under the conditions of reasonable positions. Otherwise, if the position is too light, the retracement will be small, but the income will also be reduced, and the profit and loss will come from the same source, and it will be a pity to miss the opportunity; if the position is too heavy, it will be a mistake, and sooner or later it will be finished.
However, it is recommended that novices start with a real-time implementation of this strategy. Only through practice can you quickly improve your trading level. But be sure to keep your position light, and the leverage should be lower. It is best not to use leverage, so that you can persist for a longer period of time.
If you want to be more professional, reduce single-point risks, the capital curve will be smoother, and you will have less to worry about when the offer is made (as mentioned before, under reasonable positions, this is actually very important, because it is easier to stick to the strategy and wait until the risk ) is generally a multi-variety, multi-strategy, multi-parameter model, and then there are additions and subtractions of positions. It does not necessarily mean adding or subtracting positions on one strategy, but can also be achieved by multiple sub-strategies to achieve the effect of adding or subtracting positions. However, in this case, the strategy will be more crowded. If you also want to minimize the slippage of opening and closing positions, the code complexity will skyrocket.
Let’s talk about the difficulties caused by this multi-variety, multi-strategy and multi-parameter model.
difficulty
1. API frequency limit
The first difficulty with multiple strategies is the API frequency limit. There are many varieties and strategies. In order to obtain real-time prices, positions, orders and other information, you have to constantly visit the exchange. As mentioned before, if the frequency is low and slow, the obtained price may not be the latest, and the slippage may be much greater.
The server resources of the exchange are limited, so there is a limit on the access frequency of the API. Binance’s limit is 2,400 per minute, and then there is a 10-second limit, and there are other so-called machine learning modes for detecting malicious behavior. Anyway, it will definitely not work if it is too frequent. As for why it doesn’t work, the exchange has the final say. In this case, you You know, there’s no formula.
Although it is said to be 2400, I tested it casually using concurrency. If I request K lines, even if it is the shortest 99 lines at a time (API weight is 1), then if I request about 75 coins at the same time, I will be banned for a few minutes. Therefore, one cannot easily challenge the limits.
If you violate it, you will receive 429, 418 errors, and your IP will be banned for a few minutes to a few days. At this time, if you have a position, it will be miserable (there are no restrictions on placing orders, especially liquidating existing positions, but you have no price information. , unless there is also websocket)
The solution to this problem is to use websocket to obtain data, so that the exchange can actively push it over in a timely manner without occupying the API. Can reduce some slippage. The problem is that you have to maintain another websocket-based market center. The difficulty of live code has increased and become more complicated. It must not be disconnected for 24 hours, and if it is disconnected, it must be automatically reconnected in time, etc.
2. Strategy status
As mentioned before, if the strategies become more complex, you have to record the status of each strategy. Otherwise, the exchange's positions will not know which strategy opened, how much each opened, etc. And if you want to review it later, you will have to record more data.
At this time, you have to introduce a database (of course, you can also use file records, the same idea).
The introduction of a database brings about the problems of the database. However, this is similar to software development in other industries and has nothing new. Especially in the e-commerce industry, because it is all about fund balances, order management and other similar things, those who have no experience can read such books and articles.
It should be noted that many steps need to achieve atomic operations similar to database transactions, that is, a set of operations or all of them succeed. Once a certain step fails, you have to do nothing and roll back to the original state.
Then, because it is a real money transaction after all, for database operations of order information, it is best to use the highest level of the four isolation levels, Serializable, to eliminate the possibility of all dirty reads, non-repeatable reads, and phantom reads. . That is to say, no concurrency, multi-threading, or even asynchronous use is used to read and write key information of the database. All operations are completed in one thread. Implementation of operations.
Don't take the challenge of concurrent programming lightly. If there is a problem with this kind of system, it is probably impossible to fix it. It is the most difficult bug to find in software development. Those who can complete this kind of work neatly are programmers with a monthly salary of 50,000+.
There are not many that I can think of that use multi-threading or asynchronous (multi-threading and asynchronous, it is recommended to use multi-threading, because asynchronous is like an infectious disease in Python. Once used, it must be used in a call chain) What is really beneficial is sending DingTalk warning signals and the like. Because the DingTalk service is in China, and the currency exchange servers are overseas, your code server should be deployed overseas, closer to the exchange, so when sending DingTalk, sometimes it takes several seconds to return, and sometimes it is even blocked. It lasts for more than ten seconds.
at last
In fact, it doesn’t matter if you can’t do many things. As long as you have the awareness to accept a larger slippage, it is possible to spit out almost 1/5 or even more profits. The most important thing is the strategy. Expecting positive returns and being able to run without any downtime is the key.
In short, the first priority of a firm offer is to continue and stably execute the established logic of the strategy. Slippage is part of trading, just try to reduce it as long as it doesn't break your muscles.
First-class strategy, 2 and 3 streams are implemented, there is no big problem. But if the strategy is not up to standard, the top real trading code will not help, and the loss will still be a loss. Therefore, just writing good code cannot do quantitative trading well. The strategy is good and the live code is robust. Of course it is the best. However, when time is limited, focus on strategy.
To be continued
