Can We Lastly Use ChatGPT as a Quantitative Analyst?
In two of our earlier articles, we explored the concept of utilizing synthetic intelligence to backtest buying and selling methods. Since then, AI has continued to develop, with instruments like ChatGPT evolving from easy Q&A assistants into extra complicated instruments which will assist in growing and testing funding methods—at the least, in line with among the extra optimistic voices within the subject. Over a 12 months has handed since our first experiments, and with all the present hype across the usefulness of enormous language fashions (LLMs), we consider it’s the appropriate time to critically revisit this matter. Due to this fact, our purpose is to judge how effectively in the present day’s AI fashions can carry out as quasi-junior quantitative analysts—highlighting not solely the promising use instances but additionally the constraints that also stay.
Mannequin choice
First, we wanted to pick a mannequin appropriate for the duty. We explored the choices of utilizing Claude AI, Gemini Superior (previously Deep Analysis), and ChatGPT, as these are among the most generally used AI instruments in the present day. Progress within the AI fashions goes actually quick; a few of them are higher in selective sub-tasks, and others are worse; nonetheless, from our perspective, we now have not seen important variations between them. Due to this fact, primarily based on our wants – knowledge imputation, code interpretation, and reasoning, we selected ChatGPT as a main device wherein carried out our evaluation. When deciding which particular model to make use of, we chosen the GPT-4o mannequin, because it proved to be probably the most versatile total. We additionally thought of the GPT-4.5 mannequin (which is marketed by the OpenAI as higher mannequin for analytical duties), however since it’s anticipated to be depreciated quickly, we felt this text wouldn’t supply lasting relevance if primarily based on it.
What we wish to accomplish
Because the title of this text suggests, our purpose was to search out out whether or not the method of making a buying and selling technique may be assisted by AI, or if not the entire course of, then if at the least some a part of the method may be outsourced to the AI and and if we nonetheless can belief the outcomes. For that, we determined to stay to the easy mannequin – we labored with ChatGPT and requested him to help us in creating an asset allocation technique utilizing three property – equities, mounted revenue and commodities.
Our exams have been carried out on knowledge from 07.07.2015 to 17.04.2025 for SPY (SPDR S&P 500 ETF Belief), IEF (iShares 7-10 Yr Treasury Bond ETF) and DBC (Invesco DB Commodity Index Monitoring Fund) as funding universe.
First iterations
When the information have been ready (we bumped into some points, however we’ll summarize them later), implementing a easy buying and selling technique, like fixed-percentage allocation, was a comparatively simple activity. Easy methods contain assigning a set portion of capital to totally different property, no matter market situations. For instance, you may allocate 60% to shares, 30% to bonds, and 10% to commodities. In code, this simply means multiplying every asset’s return by its goal weight and summing them as much as get the portfolio return. You don’t want complicated indicators or dynamic rebalancing, simply fundamental arithmetic operations on time sequence knowledge. This sort of technique is right for the beginning of AI automation and testing as a result of the logic is simple and may be utilized persistently over the dataset.
The AI mannequin additionally does somewhat bit extra. Not solely can it write code for such a fundamental technique, however it will probably counsel a few of them by itself. Due to this fact, we began with a naive technique and requested AI to counsel us modification of allocation ratios, that are rational and affordable and counsel us methods, which can be extra worthwhile when it comes to returns, Sharpe ratio and Calmar ratio.
Options
After operating the essential mounted asset allocation methods and checking their efficiency, the subsequent step was clear: can we do higher? It’s one factor to create a easy portfolio with mounted weights, however markets are not often that cooperative. So we requested ChatGPT not simply to check the naive technique (and variations) but additionally to assist provide you with affordable modifications that may enhance the outcomes with out making the entire thing overly difficult.
That is the place issues get extra fascinating. As a substitute of simply assigning static weights, we explored small variations: what occurs if we shift a bit extra into bonds throughout tough intervals or barely enhance fairness publicity in robust uptrends? We intentionally prevented leaping into complicated machine-learning fashions or regime-switching strategies. The purpose right here was modest – introduce simply sufficient construction to mirror real-world pondering, like adapting to latest efficiency or volatility. ChatGPT might deal with that, (as soon as once more, not with out issues), however ultimately, it was in a position to counsel methods to re-weight the portfolio or apply fundamental filters to keep away from main drawdowns. On account of these prompts, we obtained the next fairness curves:
Combining and optimising
As soon as we noticed that lively asset allocation methods might enhance efficiency, the subsequent problem was to discover a extra balanced technique – one which not solely performs effectively on paper but additionally feels strong and wise. It’s simple to get caught up in tuning parameters and selecting the most effective interval for indicators to squeeze out a barely larger Sharpe ratio, however there’s at all times a trade-off. A technique that appears nice in a single interval may crumble in one other.
To discover this, we requested ChatGPT to assist us take a look at totally different variations of the technique by adjusting key parameters – in our case, largely timeframes. The thought wasn’t to blindly optimize for the most effective outcome however to know how delicate the technique is to adjustments. If small shifts in a parameter result in huge swings in efficiency, that’s a pink flag.
Last iteration of the Asset Allocation Technique In accordance with ChatGPT is as follows:
Described technique has the next properties:
And these are fairness curves of steered lively methods and closing technique (brown):
And right here is the results of the AI performing the robustness exams to be sure that the parameter home windows we used, like lookback intervals or rebalancing intervals, weren’t simply conveniently chosen values that occurred to supply distinctive outcomes by probability.
What went good
Thus far, it appears a cheerful story, proper? We requested ChatGPT for the technique, and ultimately, we bought one. It’s undoubtedly a big improve once we evaluate the entire course of with the evaluation we carried out roughly 18 months in the past. ChatGPT orientates itself effectively in quant finance and might counsel a number of variations for the asset allocation methods after which at all times provide you with options for the subsequent steps within the evaluation. The exploratory a part of the quant evaluation is well-handled. ChatGPT is an AI chatbot, and as such, it will probably talk a number of concepts and focus on them eloquently.
Nevertheless, right here comes the catch – it’s nonetheless a chatbot, not an information analyst, and the chatbot’s main focus is to make you pleased with the “chatting.” What does it imply? It tends to be over-optimistic and sycophantic – it doesn’t “assume”, it solutions questions and tries to make you prepared to proceed within the dialog. A whole lot of the time, ChatGPT offered its concepts or evaluation and made extraordinarily naive errors in it; nonetheless, it offered outcomes as the most effective technique/concept ever in existence. The fixed re-checking of the person steps within the evaluation was actually tiring.
What went unsuitable
So, what have been the problems we encountered, and what must you take note of while you experiment with chatbots as assistants in quantitative finance?
Knowledge preparation
We encountered a couple of points when working with knowledge. Initially, we tried to acquire the information instantly from the web by way of ChatGPT, however that wasn’t possible-so we had to offer the information ourselves. This led to some sudden issues. Since we used dates within the format DD.MM.YYYY and numbers with a comma because the decimal separator, ChatGPT actually struggled to interpret the information appropriately. Probably the most dependable strategy turned out to be offering the information in a format that ChatGPT is extra acquainted with-typically utilizing YYYY-MM-DD for dates and a dot because the decimal level. Getting ready the dataset on this manner will make the interplay smoother and scale back misunderstandings throughout evaluation.
Knowledge corruption
After operating a number of fashions on the inputted dataset, we skilled a couple of points. In some instances, the order of the information modified unexpectedly; in others, complete sections of knowledge have been misplaced. This led to outputs that have been clearly incorrect or inconsistent with what we anticipated. The outcomes appeared like this:
This difficulty is carefully associated to how reminiscence works when dealing with our knowledge. We often needed to re-upload the identical dataset, because it was both forgotten through the evaluation course of or grew to become corrupted in varied methods (and we didn’t perceive the explanation for corruption). This may make it tougher sooner or later to keep up consistency throughout exams and highlights the constraints of working with bigger datasets in this sort of setup.
Ultimately, if you want to do your individual take a look at evaluation, we might undoubtedly suggest offering a chatbot with your individual knowledge. As ChatGPT tends to make errors within the preliminary knowledge dealing with, should you depend on the information from ChatGPT itself, you wouldn’t be capable to catch among the errors it makes.
Want for validation
When utilizing AI to create a method, you usually wish to plot fairness curves, calculate fundamental efficiency metrics, and so forth. Nevertheless, the mannequin might interpret these duties in its personal manner, which doesn’t at all times match your expectations. Typically the problems are apparent at first look, however extra usually, you must examine the code rigorously. The commonest errors often happen in knowledge formatting, the implementation of the technique operate, and the way returns, danger, and drawdowns are calculated.
One other associated difficulty is overpromising on the theoretical facet whereas underdelivering within the precise code. This usually implies that the mannequin describes, for instance, a method consisting of three guidelines utilized to a dataset, however solely implements two of them. In our case, the technique was supposed to include momentum, volatility, and correlations. Nevertheless, correlations weren’t used within the implementation.
Hallucinations
Within the context of AI, it sometimes refers to when a mannequin generates info that’s factually incorrect or fabricated, although it might sound believable.
ChatGPT
In our case, we have been exploring a number of methods directly and aimed to research simply the efficiency of probably the most profitable amongst them. This setup elevated the danger of errors going unnoticed-especially when the mannequin appeared to execute every step appropriately, however had truly skipped or misapplied components of the technique logic. With out cautious evaluation, these inconsistencies can result in deceptive conclusions a couple of technique’s effectiveness.
After we obtained the code for this technique and ran our personal evaluation, the outcomes we bought have been considerably totally different.
After importing the information into the mannequin a second time, the outcomes it produced matched our personal. How the ChatGPT calculated higher ratios within the first time? And why have been they totally different? We do not know.
This introduced us again to an vital a part of the method – we (customers, people) need to validate leads to every step of the evaluation. Irrespective of how small or insignificant step it appears. It’s completely essential. ChatGPT generally produces completely made-up numbers (even when the code it suggests for calculation of these numbers is right).
Cyclic conversations
After we found errors within the calculated efficiency metrics, we needed to know why they occurred. After a couple of follow-up prompts, the mannequin circled round varied explanations-differences in knowledge, discrepancies within the construction of the technique, or changes to its parameters. Nevertheless, we identified (appropriately) that none of those utilized, since we had merely run the precise code offered by ChatGPT on the identical dataset we had initially equipped. Even after asking the mannequin to re-run its code on the identical enter, we discovered ourselves in a loop, the place the AI continued to deflect the difficulty fairly than acknowledge or right the defective calculations. This expertise illustrates a key limitation of utilizing AI to debug or take a look at a method: whereas it might appear assured, it doesn’t at all times reliably hint the basis of its personal errors.
If we take a step again and use AI only for brainstorming technique concepts, we might encounter the same difficulty. The mannequin usually will get caught on one fundamental idea and tends to construct all the pieces round it. For instance, if we start with a method that entails deciding on the highest N property primarily based on a sure criterion, the mannequin might proceed to counsel solely variations that deal with this choice step as important. Except we explicitly state that we wish to keep away from utilizing that criterion, it can possible stay a core a part of each new proposal. This highlights a standard limitation: AI tends to anchor on the preliminary route and struggles to discover fully totally different concepts except firmly guided to take action.
Tendention to over-optimization
ChatGPT, as an analyst, tends to be an optimization machine. Options it offers, or concepts it presents as worthwhile to research have a tendency so as to add levels of freedom into the technique, and as such, the technique turns into increasingly over-optimized to the previous knowledge. ChatGPT doesn’t generalize effectively (as of now) and often picks the best-performing model of the technique after which appears to be like out for the reason of why it’s the most effective and tries to enhance it much more. It’s logical (from the chatbot’s perspective), but it surely’s not the most effective concept if you wish to construct a strong buying and selling technique. Due to this fact, usually, ChatGPT’s options have a restricted worth, and it’s often higher to immediate it to proceed in numerous instructions than it suggests. All in all, it’s higher when a human is in cost than relying blindly on a chatbot throughout evaluation.
Conclusion
Synthetic intelligence is a robust device that may help with many duties. It’s good at suggesting top-down concepts, drafting code outlines for testing, and sometimes serving to you discover a new route while you’re caught on an issue. Nevertheless, there are a number of vital limitations to bear in mind. As an example, you continue to have to supply your individual knowledge for evaluation, rigorously test the code for lots of potential errors, and keep away from totally trusting the efficiency metrics (and even charts) printed by the mannequin with out verification.
Since our earlier article, AI has made important progress. What it will probably do is assist automate components of the workflow and avoid wasting valuable time. Nevertheless, even with these developments, the potential for errors stays excessive. That’s a danger that must be calculated while you attempt to work with it. AI is a classical device, like a pointy knife – you can also make a number of helpful issues with it, or should you have no idea what you’re doing, then you possibly can minimize your individual finger with it.
Authors: David Belobrad, Quant Analyst, QuantpediaRadovan Vojtko, Head of Analysis, Quantpedia
Are you on the lookout for extra methods to examine? Join our e-newsletter or go to our Weblog or Screener.
Do you wish to be taught extra about Quantpedia Premium service? Examine how Quantpedia works, our mission and Premium pricing supply.
Do you wish to be taught extra about Quantpedia Professional service? Examine its description, watch movies, evaluation reporting capabilities and go to our pricing supply.
Are you on the lookout for historic knowledge or backtesting platforms? Examine our listing of Algo Buying and selling Reductions.
Would you want free entry to our companies? Then, open an account with Lightspeed and luxuriate in one 12 months of Quantpedia Premium without charge.
Or observe us on:
Fb Group, Fb Web page, Twitter, Linkedin, Medium or Youtube
Share onLinkedInTwitterFacebookConfer with a pal
