Incorporating real-world parameters into simulated(paper) trading
"I think you're on the wrong track.  Getting more and more samples from the real world does not make your backtest more accurate, it just confirms that your strategy can withstand one particular sample path of a stochastic process. The reason why you find it simple to incorporate fees, commissions, taxes, etc. is because they're a static and constant process -- well they might change over time but most definitely uncorrelated to the markets. Modelling overnight returns or the top levels of the order book the next day is serious work.  First you have to select a suitable model (that's mostly theoretical work but experience can help a lot).  Then, in order to do it data-driven, you'd have to plough through thousands of days of sample data on a set of thousands of instruments to get a ""feeling"" (aka significant model parameters).   Apropos data mining, I think Excel might be the wrong tool for the job.  Level-2 data (even just the first 10 levels) is a massive blob.  For example, the NYSE OpenBook historical data weighs in at a massive 15 TB compressed (uncompressed 74 TB) for the last 10 years, and costs USD 200k.  Anyway, as for other factors to take into account: So how to account for all this in a backtest?  Personally, I would put in some penalty terms (as % on a return basis) for every factor you want to consider, don't hardcode them.  You can then run a stress test by exploring these parameters (i.e. assign some values in the range of 0 to whatever fits).  Explore them individually (only set one penalty term at a time) to get a feeling how the strategy might react to stress from that factor.  Then you can run the backtest with typical (or observed) combinations of penalty factors and slowly stress them altogether. Edit Just to avoid confusion about terminology.  A backtest in the strict sense (had I implemented this strategy X years ago, what would have happened?) won't benefit from any modelling simply because the real-world ""does the sampling"" for us. However, to evaluate a strategy's robustness you should account for the additional factors and run some stress tests.  If the strategy performs well in the real-world or no-stress scenario but produces losses once a tiny slippage occurs every now and again, you could conclude that the strategy is very fragile.  The key is to explore the maximum stress the strategy can handle (by whatever measure); if a lot you can call the strategy robust. The latter is what I personally call a backtest; the first procedure would go by the name ""extension towards the past"" or so. Some lightweight literature:" You said the decision will be made by EOD. If you've made the decision prior to the market close, I'd execute on the closing price.  If you are trading stocks with any decent volume, I'd not worry about the liquidity.  If your strategy's profits are so small that your gains are significantly impacted by say, the bid/ask spread (a penny or less for liquid stocks) I'd rethink the approach. You'll find the difference between the market open and prior night close is far greater than the normal bid/ask.
You can try paper trading to sharpen your investing skills(identifying stocks to invest, how much money to allocate and stuff) but nothing compares to getting beaten black and blue in the real world. When virtual money is involved you mayn't care, because you don't loose anything, but when your hard earned money disappears or grows, no paper trading can incite those feelings in you. So there is no guarantee that doing paper trading will make you a better investor, but can help you a lot in terms of learning. Secondly educate yourself on the ways of investing. It is hard work and realize that there is no substitute for hard work. India is a growing economy and your friends maybe safe in the short term but take it from any INVESTOR, not in the long run. And moreover as all economies are recovering from the recession there are ample opportunities to invest money in India both good and bad. Calculate your returns and compare it with your friends maybe a year or two down the lane to compare the returns generated from both sides. Maybe they would come trumps but remember selecting a good investment from a bad investment will surely pay out in the long run.  Not sure what you do not understand what Buffet says. It cannot get more simpler than that. If you can drill those rules into your blood, you mayn't become a billionaire but surely you will make a killing, but in the long run. Read and read as much as you can. Buy books, browse the net. This might help. One more guy like you. The papers you would need to buy are called 'futures', and they give you the right to buy (or sell) a certain amount of oil at a certain location (some large harbor typically), for a certain price, on a certain day. You can typically sell these futures anytime (if you find someone that buys them), and depending on the direction you bought, you will make or lose money according to oil rice changes - if you have the future to get oil for 50 $, and the market price is 60, this paper is obviously worth 10 $. Note that you will have to sell the future at some day before it runs out, or you get real oil in some harbor somewhere for it, which might not be very useful to you. As most traders don't want really any oil, that might happen automatically or by default, but you need to make sure of that. Note also that worst case you could lose a lot more money than you put in - if you buy a future to deliver oil for 50 $, and the oil price runs, you will have to procure the oil for new price, meaning pay the current price for it. There is no theoretical limit, so depending on what you trade, you could lose ten times or a thousand times what you invested. [I worded that without technical lingo so it is clear for beginners - this is the concept, not the full technical explanation] While trading in stochastic I've understood, one needs reference (SMA/EMA/Bolinger Band and even RSI) to verify trade prior entering it. Stochastic is nothing to do with price or volume it is about speed. Adjusting K% has ability to turn you from Day trader to -> swing trader to -> long term investor. So you adjust your k% according to chart time-frame. Stochastic setup for 1 min, 5 min ,15, 30, 60 min, daily, weekly, monthly, quarterly, half yearly and yearly are all different. If you try hopping from one time-frame to another just because it is below oversold or above overbought region with same K%, you may get confused. Worst you may not square-off your loss making trade. And rather not use excel; charts gives better visual for oscillators. "You're mostly correct, although I think you're missing something essential about no-arbitrage versus an arbitrage argument.  Black-Scholes makes an arbitrage argument, which is that the value of an option should be the same as any portfolio that has identical cash flows, and this is generally a sound argument.  Notice, however, that BS is ultimately an equilibrium model: it tells you the ""correct"" price of an option if the assumptions of BS hold, and doesn't necessarily match observed market prices.  A no-arbitrage condition or model deliberately incorporates observed prices (or yields, or whatever) into the model, so that there cannot be an arbitrage opportunity implied by the model.  This comes up a lot in term structure of interest rate problems, where equilibrium models like Vasicek or Cox-Ingersoll-Ross won't perfectly reproduce the current, observed term structure, and so imply an arbitrage opportunity.  No-arbitrage models like Hull-White specifically match the model's term structure to observed yields/prices, so that there is no arbitrage opportunity between the observed term structure and your model of it.  It's important to note that this will still allow for arbitrage involving bonds that are *not* part of the observed term structure.  As for equivalent martingale measures, you might think about it more generally.  The process involves changing the probability distribution from the actual (which is hard to use for pricing) to a different one that's easier to use but will result in the same prices; this is nearly always a risk-neutral probability.  You can think of equivalent martingale  pricing as asking, ""how would this security behave, and be priced, in a world that is completely risk neutral"", and then making an argument that the prices are in fact equivalent.  EDIT: grammar" Machine learning is definitely applied to trading, but I have not tried it myself. For now I've been focused on figuring out the platforms and how they work; I have not been trying out other strategies besides a SMAC strategy. The most machine learning-like application I've attempted was cross-validation by walk-forward analysis (I'm publishing that post on Monday).  I know nothing about TensorFlow other than it's used for deep learning and that it doesn't work on Windows and thus would not work on my more powerful gaming computer, and like I said above, I have not been exploring machine learning right now. Neural networks are on my radar, on the list of things I need to read, but there was a topic on r/algotrading recently where most users said that deep learning has not demonstrated better performance than more traditional ML techniques and looks like a fad. I want to convince myself of that first, though.  I'm glad you enjoyed the post and my site! Thank you! This paper makes the rounds every so often and I HATE IT VERY MUCH.  No one has EVER EVER EVER estimated standard deviation by taking an average of abs deviations, because that would be stupid, further, realized vol is a very different beast than implied vol. AND FURTHER, the models are fitted to the data, rather than fighting the tape and fitting the data to the models.   Most people working with these instruments are trying to figure out the prices of unknown derivatives based on known derivatives - the models just help interpret findings and estimate what's necessary to hedge a position. thanks for the advice, I still have few weeks before my master course starts and would like to do more reading regarding trading, any books that you would recommend ?   Also I always assume that modelling skill is not that important in sales and trading, is my assumption correct ? Modelling is probably one of my weak skills but if it is needed I would like to work further on it   thank you "That's the way the markets work in THEORY. In actual fact, markets are subject to ""real world"" pressures. That is, there are so many things going on in the market that the end of the ""Lined In"" lock up is just one of many. To produce the result you describe, traders would have to hold cash in reserve for this so-called ""contingency"" to buy at the end of the lock-up. In most cases, they wouldn't want to because of everything else that is going on.  To use a real world analogy, would you want to wait until the last possible moment before going to the bathroom? Or would you go now while you had the chance? That's what the decision about ""holding cash in reserve for a contingency"" is like." "Check out Professor Damodaran's website: http://pages.stern.nyu.edu/~adamodar/ . Tons of good stuff there to get you started. If you want more depth, he's written what is widely considered the bible on the subject of valuation: ""Investment Valuation"". DCF is very well suited to stock analysis. One doesn't need to know, or forecast the future stock price to use it. In fact, it's the opposite. Business fundamentals are forecasted to estimate the sum total of future cash flows from the company, discounted back to the present. Divide that by shares outstanding, and you have the value of the stock. The key is to remember that DCF calculations are very sensitive to inputs. Be conservative in your estimates of future revenue growth, earnings margins, and capital investment. I usually develop three forecasts: pessimistic, neutral, optimistic. This delivers a range of value instead of a false-precision single number. This may seem odd: I find the DCF invaluable, but for the process, not so much the result. The input sensitivity requires careful work, and while a range of value is useful, the real benefit comes from being required to answer the questions to build the forecast. It provides a framework to analyze a business. You're just trying to properly fill in the boxes, estimate the unguessable. To do so, you pore through the financials. Skimming, reading with a purpose. In the end you come away with a fairly deep understanding of the business, how they make money, why they'll continue to make money, etc." Quick and dirty paper but pretty interesting.. I'm not in Portfolio Management but I probably would have ended up at the modal number as well.   I don't know the subject deeply enough to answer my own question, but is the bias always toward underestimation of variance? Or is that a complex of the way the problem was set up?   Another question I have for those in investment management; Would this impact asset allocation? In a simple world yes, but not in the real world. Option pricing isn't that simplistic in real life. Generally option pricing uses a Monte Carlo simulation of the Black Scholes formula/binomial and then plot them nomally to decide the optimum price of the option. Primarily multiple scenarios are generated and under that specific scenario the option is priced and then a price is derived for the option in real life, using the prices which were predicted in the scenarios.  So you don't generate a single price for an option, because you have to look into the future to see how the price of the option would behave, under the real elements of the market. So what you price is an assumption that this is the most likely value under my scenarios, which I predicted into the future. Because of the market, if you price an option higher/lower than another competitor you introduce an option for arbitrage by others. So you try to be as close to the real value of the option, which your competitor also does. The more closer your option value is to the real price the better it is for all. Did you try the book from Hull ? EDIT: While pricing you generally take variables which would affect the price of your option. The more variables you take(more nearer you are to the real situation) the more realistic your price will be and you would converge on the real price faster. So simple formula is an option, but the deviations maybe large from the real value. And you would end up loosing money, most of the time. So the complicated formula is there for getting a more accurate price, not to confuse people. You can use your formula, but there will be odds stacked against you to loose money, from the onset, because you didn't consider the variables which might/would affect the price of your option. For starters, that site shows the first 5 levels on each side of the book, which is actually quite a bit of information.  When traders say the top of the book, they mean just the first level.  So you're already getting 8 extra levels. If you want all the details, you must subscribe to the exchange's data feeds (this costs thousands of dollars per month) or open an account with a broker who offers that information. More important than depth, however, is update frequency.  The BATS site appears to update every 5 seconds, which is nowhere near frequently enough to see what's truly going on in the book.  Depending on your use case, 2 levels on each side of the book updated every millisecond might be far more valuable than 20 levels on each side updated every second. "The previous answers make valid points regarding the risks, and why you can't reasonably compare trading for profit/loss to a roll of the die. This answer looks at the math instead. Your assumption: I have an equal probability to make a profit or a loss. Is incorrect, for the reasons stated in other answers. However, the answer to your question: Can I also assume that probabilistically speaking, a trader cannot do worst than random? Is ""yes"". But only because the question is flawed. Consequently it's throwing people in all directions with their answers. But quite simply, in a truly random environment the worst case scenario, no matter how improbable, is that you lose over and over again until you have nothing left. This can happen in sequential rolls of the dice AND in trading securities/bonds/whatever. You could guess wrong for every roll of the die AND all of your stock picks could become worthless. Both outcomes result in $0 (assuming you do not gamble with credit). Tell me, which $0 is ""worse""? Given the infinite number of plays that ""random"" implies, the chance of losing your entire bankroll exists in both scenarios, and that is enough by itself to make neither option ""worse"" than the other. Of course, the opposite is also true. You could only pick winners, with an unlimited upside potential, but again that could happen with either dice rolls or stock picks. It's just highly improbable. my chances cannot be worse than random and if my trading system has an edge that is greater than the percentage of the transaction that is transaction cost, then I am probabilistically likely to make a profit? Nope. This is where it all falls apart. Just because your chances of losing it all are similarly improbable, does not make you more likely to win with one method or the other. Regression to the mean, when given infinite, truly random outcomes, makes it impossible to ""have an edge"". Also, ""probabilistically"" isn't a word, but ""probably"" is." Well said.  To put it shortly I think both can be a viable source of some side income when proper risk management is in place.  It is likely not going to work when you are trading/betting with money that is important to you.  Paper trade/bet until you find a viable strategy.  Then use proper bankroll management and some expendable income to pick up some extra bucks on the side.  Sports betting is nice because the initial investment is much lower than day trading. "You make several good points.  I'll start with Black-Scholes; the arbitrage argument in Black-Scholes is between the option and a hypothetical and unobserved portfolio with identical cash flows (the replicating portfolio).  We then value the replicating portfolio via an equilibrium model.  When you use BS, there isn't necessarily *any* observed security whose is guaranteed not imply some arbitrage opportunity, because BS makes no reference to observed prices.  A no-arbitrage modification might look like this: ""I observe the prices (and implied volatilities) of some options, and use the implied volatilities to price another option (maybe with a different strike).  Doing so ensures that there is no arbitrage between my price and the market prices implied by my model.""  Realistically, the problem arbitrage-free models are addressing is that our models and assumptions are wrong, even though they're reasonable approximations a lot of the time.  A no-arbitrage model removes some set of obvious deficiencies, but at the cost of not being able to explain why things are priced as they are.  So, for instance, Vasicek won't reproduce the observed term structure, and Hull-White fixes this, but Hull-White doesn't explain where the term structure comes from (i.e., what the term structure *should* be)."