Automation of Algorithmic Trading Strategies in Artificial Financial Markets by Combining Machine Learning Techniques and Agent-based Modeling

Document Type : Original Article

Authors

1 Ph.D. Candidate in Information Technology Management, Department of Management, Isfahan (Khorasgan) Branch, Islamic Azad University, Isfahan, Iran.

2 Senior Lecturer, School of Computing, National University of Singapore, 117417, Singapore.

3 Associate Prof., Department of Economics, Isfahan (Khorasgan) Branch, Islamic Azad University, Isfahan, Iran.

4 Associate Prof., Department of Economics, Faculty of Administrative Science and Economics, University of Isfahan, Isfahan, Iran.

10.30699/ijf.2025.458379.1472
Abstract
This study aims to demonstrate the performance of algorithmic trading strategies compared to traditional trading methods in artificial financial markets. This research uses a hybrid model based on agent-based modeling and machine learning methods to simulate agents' behavior in an artificial financial market. This model includes two categories, traditional agents and intelligent agents. Traditional agents are divided into three groups: liquidity providers, liquidity consumers, and noise traders. Intelligent agents are trained using deep learning techniques and recurrent neural networks. Based on the developed algorithms, the agent-based model simulates both categories of traditional and trained agents in an artificial financial market. Sensitivity analysis tests were used to test the validity and reliability of the model, and the values of the fat-tailed distribution of returns, volatility clustering, autocorrelation of returns, long memory in order flow, concave price impact, and extreme price events are calculated in the model and compared with the standardized values. Historical data was used to predict stock prices, and model simulations were used to generate trading signals and update the limited order book. The results of executing the model show the ability of intelligent agents to trade in artificial financial markets compared to traditional agents.

Keywords


Alexander, S. S. (1961). Price movements in speculative markets: Trends or random walks. Industrial Management Review (pre-1986), 2(2), 7.
Aloud, M. E. (2020). The role of attribute selection in Deep ANNs learning framework for high‐frequency financial trading. Intelligent Systems in Accounting, Finance and Management, 27(2), 43-54.
Avellaneda, M., & Stoikov, S. (2008). High-frequency trading in a limit order book. Quantitative Finance, 8(3), 217–224.
Bodie, Z., Kane, A., & Marcus, A. J. (2018). Investments: McGraw-Hill Education.
Booth, A. (2016). Automated algorithmic trading: Machine learning and agent-based modelling in complex adaptive financial markets. University of Southampton,
Bouchaud, J.-P. (2002). An introduction to statistical finance. Physica A: Statistical Mechanics and its Applications, 313(1-2), 238-251.
Calvez, B., & Hutzler, G. (2005). Automatic tuning of agent-based models using genetic algorithms. Paper presented at the International workshop on multi-agent systems and agent-based simulation.
Chaboud, A. P., Chiquoine, B., Hjalmarsson, E., & Vega, C. (2014). Rise of the machines: Algorithmic trading in the foreign exchange market. The Journal of Finance, 69(5), 2045-2084.
Cont, R. (2001). Empirical properties of asset returns: stylized facts and statistical issues. Quantitative Finance, 1(2), 223.
Crooks, A., Heppenstall, A., Manley, E., & Malleson, N. (2018). Agent-based modelling and geographical information systems: a practical primer.
Dixon, M. (2017). Sequence classification of the limit order book using recurrent neural networks. Journal of Computational Science. doi:https://doi.org/10.1016/j.jocs.2017.08.018
Dixon, M. F. (2020). Machine Learning in Finance: from Theory to Practice: Springer Nature.
Donadio, S., & Ghosh, S. (2019). Learn Algorithmic Trading: Build and Deploy Algorithmic Trading Systems and Strategies Using Python and Advanced Data Analysis: Packt Publishing.
Evensen, G. (2003). The ensemble Kalman filter: Theoretical formulation and practical implementation. Ocean dynamics, 53, 343–367.
Fama, E. F. (1970). Efficient capital markets: A review of theory and empirical work. The Journal of Finance, 25(2), 383-417.
Fama, E. F., & Blume, M. E. (1966). Filter rules and stock-market trading. The Journal of Business, 39(1), 226–241.
Frazier, P. I. (2018). A tutorial on Bayesian optimization. arXiv preprint arXiv:1807.02811.
Gould, M. D., Porter, M. A., Williams, S., McDonald, M., Fenn, D. J., & Howison, S. D. (2013). Limit order books. Quantitative Finance, 13(11), 1709-1742.
Graves, A. (2013). Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850.
Guo, T. X. (2005). An agent-based simulation of double-auction markets: University of Toronto.
Heaton, J. B., Polson, N. G., & Witte, J. H. (2017). Deep learning for finance: deep portfolios. Applied Stochastic Models in Business and Industry, 33(1), 3–12.
Hendershott, T., Jones, C. M., & Menkveld, A. J. (2011). Does algorithmic trading improve liquidity? The Journal of Finance, 66(1), 1-33.
Hirsa, A. (2013). An introduction to the mathematics of financial derivatives: Academic press.
Hochreiter, S., Bengio, Y., Frasconi, P., & Schmidhuber, J. (2001). Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In: A field guide to dynamical recurrent neural networks. IEEE Press.
Huang, Z.-F., & Solomon, S. (2001). Power, Lévy, exponential and Gaussian-like regimes in autocatalytic financial systems. The European Physical Journal B-Condensed Matter and Complex Systems, 20(4), 601–607.
Jacob Leal, S., Napoletano, M., Roventini, A., & Fagiolo, G. (2016). Rock around the clock: An agent-based model of low-and high-frequency trading. Journal of Evolutionary Economics, 26, 49-76.
Johnson, B. (2010). Algorithmic Trading & DMA: An introduction to direct access trading strategies (Vol. 1): 4Myeloma Press London.
Johnson, N., Zhao, G., Hunsader, E., Qi, H., Johnson, N., Meng, J., & Tivnan, B. (2013). Abrupt rise of new machine ecology beyond human response time. Scientific reports, 3(1), 1–7.
Kanzari, D., & Said, Y. R. B. (2023). A complex adaptive agent modeling to predict the stock market prices. Expert Systems with Applications, 222, 119783.
Kim, G.-r., & Markowitz, H. M. (1989). Investment rules, margin, and market volatility. Journal of portfolio management, 16(1), 45.
Levy, M., Levy, H., & Solomon, S. (1994). A microscopic stock market model: cycles, booms, and crashes. Economics Letters, 45(1), 103–111.
Lillo, F., & Farmer, J. D. (2004). The long memory of the efficient market. Studies in nonlinear dynamics & econometrics, 8(3).
Lillo, F., Farmer, J. D., & Mantegna, R. N. (2003). Master curve for price-impact function. Nature, 421(6919), 129–130.
McGroarty, F., Booth, A., Gerding, E., & Chinthalapati, V. R. (2019). High frequency trading strategies, market fragility and price spikes: an agent based model perspective. Annals of Operations Research, 282(1), 217–244.
Oesch, C. (2014). An agent-based model for market impact.
Peng, C. K., Buldyrev, S. V., Havlin, S., Simons, M., Stanley, H. E., & Goldberger, A. L. (1994). Mosaic organization of DNA nucleotides. Physical review e, 49(2), 1685.
Samanidou, E., Zschischang, E., Stauffer, D., & Lux, T. (2007). Agent-based models of financial markets. Reports on Progress in Physics, 70(3), 409.
Sobol, I. M. (2001). Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Mathematics and computers in simulation, 55(1-3), 271-280.
Taylor, G. W., & Hinton, G. E. (2009). Factored conditional restricted Boltzmann machines for modeling motion style—paper presented at the Proceedings of the 26th annual international conference on machine learning.
van der Hoog, S. (2017). Deep Learning in (and of) Agent-Based Models: A Prospectus. arXiv preprint arXiv:1706.06302.
Wilensky, U., & Rand, W. (2015). An introduction to agent-based modeling: modeling natural, social, and engineered complex systems with NetLogo: MIT Press.