chapter 04 revised

Author: stefan-jansen

01_machine_learning_for_trading/README.md

@@ -169,6 +169,15 @@ ML extracts signals from a wide range of market, fundamental, and alternative da
 - [](http://deeplearning.ai/), Andrew Ng
     - Andrew Ng’s introductory deep learning course
 
+### ML Competitions & Trading
+
+- [IEEE Investment Ranking Challenge](https://www.crowdai.org/challenges/ieee-investment-ranking-challenge)
+    - [Investment Ranking Challenge : Identifying the best performing stocks based on their semi-annual returns](https://arxiv.org/pdf/1906.08636.pdf)
+- [Two Sigma Financial Modeling Challenge](https://www.kaggle.com/c/two-sigma-financial-modeling)
+- [Two Sigma: Using News to Predict Stock Movements](https://www.kaggle.com/c/two-sigma-financial-news)
+- [The Winton Stock Market Challenge](https://www.kaggle.com/c/the-winton-stock-market-challenge)
+- [Algorithmic Trading Challenge](https://www.kaggle.com/c/AlgorithmicTradingChallenge)
+   
 ### Python Libraries
 
 - matplotlib [docs]( <https://github.com/matplotlib/matplotlib)

04_alpha_factor_research/00_alpha_factors_in_practice/feature_engineering.ipynb

@@ -1,25 +1,25 @@
-# Chapter 04: Alpha Factor Research & Evaluation
+# Chapter 04: Financial Feature Engineering: How to research Alpha Factors
 
 Alpha factors aim to predict the price movements of assets in the investment universe based on the available market, fundamental, or alternative data. A factor may combine one or several input variables, but assumes a single value for each asset every time the strategy evaluates the factor. 
 
 Trade decisions typically rely on relative values across assets. Trading strategies are often based on signals emitted by multiple factors, and we will see that machine learning (ML) models are particularly well suited to integrate the various signals efficiently to make more accurate predictions.
 
 This chapter provides a framework for understanding how factors work and how to measure their performance, for example using the information coefficient (IC). It demonstrates how to engineer alpha factors from data using Python libraries offline and on the Quantopian platform. It also introduces the `zipline` library to backtest factors and the `alphalens` library to evaluate their predictive power. More specifically, this chapter covers:
 
-- How to characterize, justify and measure key types of alpha factors
-- How to create alpha factors using financial feature engineering
-- How to use `zipline` offline to test individual alpha factors
-- How to use `zipline` on Quantopian to combine alpha factors and identify more sophisticated signals
+- Which categories of factors exist, why they work and how to measure them
+- How to create alpha factors using numpy, pandas, and talib
+- How to denoise data using wavelets and the Kalman filter
+- How to use zipline offline and on Quantopian to test individual and multiple alpha factors
 - How the information coefficient (IC) measures an alpha factor's predictive performance
-- How to use `alphalens` to evaluate predictive performance and turnover
+- How to use alphalens to evaluate predictive performance and turnover using, among other metrics, the information coefficient (IC)
 
-## Engineering Alpha Factor
+## Alpha Factors in practice: from data to signals
 
 Alpha factors are transformations of market, fundamental, and alternative data that contain predictive signals. They are designed to capture risks that drive asset returns. One set of factors describes fundamental, economy-wide variables such as growth, inflation, volatility, productivity, and demographic risk. Another set consists of tradeable investment styles such as the market portfolio, value-growth investing, and momentum investing.
 
 There are also factors that explain price movements based on the economics or institutional setting of financial markets, or investor behavior, including known biases of this behavior. The economic theory behind factors can be rational, where the factors have high returns over the long run to compensate for their low returns during bad times, or behavioral, where factor risk premiums result from the possibly biased, or not entirely rational behavior of agents that is not arbitraged away.
 
-### Important Factor Categories
+### On the shoulders of giants: meet the factor establishment
 
 In an idealized world, categories of risk factors should be independent of each other (orthogonal), yield positive risk premia, and form a complete set that spans all dimensions of risk and explains the systematic risks for assets in a given class. In practice, these requirements will hold only approximately.
 
@@ -31,9 +31,36 @@ In an idealized world, categories of risk factors should be independent of each
 - [Anomalies and Market Efficiency](https://www.nber.org/papers/w9277.pdf) by G. William Schwert25 (Ch. 15 in Handbook of the- "Economics of Finance", by Constantinides, Harris, and Stulz, 2003)
 - [Investor Psychology and Asset Pricing](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=265132), by David Hirshleifer (2001)
 
-### How to transform Data into Factors
+## Engineering alpha factors that predict returns
+
+
+
+### How to engineer factors using pandas and NumPy
 
 - The notebook [feature_engineering.ipynb](00_data/feature_engineering.ipynb) in the [data](00_data) directory illustrates how to engineer basic factors.
+- [Fama French](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html) Data Library
+- [numpy](https://numpy.org/) website
+    - [Quickstart Tutorial](https://numpy.org/devdocs/user/quickstart.html)
+- [pandas](https://pandas.pydata.org/) website
+    - [User Guide](https://pandas.pydata.org/docs/user_guide/index.html)
+    - [10 minutes to pandas](https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html)
+    - [Python Pandas Tutorial: A Complete Introduction for Beginners](https://www.learndatasci.com/tutorials/python-pandas-tutorial-complete-introduction-for-beginners/)
+- [alphatools](https://github.com/marketneutral/alphatools) - Quantitative finance research tools in Python
+- [mlfinlab](https://github.com/hudson-and-thames/mlfinlab) - Package based on the work of Dr Marcos Lopez de Prado regarding his research with respect to Advances in Financial Machine Learning
+
+#### How to denoise your Alpha Factors with the Kalman Filter
+
+- [PyKalman](https://pykalman.github.io/) documentation
+- [Tutorial: The Kalman Filter](http://web.mit.edu/kirtley/kirtley/binlustuff/literature/control/Kalman%20filter.pdf)
+- [Understanding and Applying Kalman Filtering](http://biorobotics.ri.cmu.edu/papers/sbp_papers/integrated3/kleeman_kalman_basics.pdf)
+- [How a Kalman filter works, in pictures](https://www.bzarg.com/p/how-a-kalman-filter-works-in-pictures/)
+
+#### How to preprocess your noisy signals using Wavelets
+
+- [PyWavelets](https://pywavelets.readthedocs.io/en/latest/) - Wavelet Transforms in Python
+- [An Introduction to Wavelets](https://www.eecis.udel.edu/~amer/CISC651/IEEEwavelet.pdf) 
+- [The Wavelet Tutorial](http://web.iitd.ac.in/~sumeet/WaveletTutorial.pdf)
+- [Wavelets for Kids](http://www.gtwavelet.bme.gatech.edu/wp/kidsA.pdf)
 
 #### References
 
@@ -44,11 +71,15 @@ In an idealized world, categories of risk factors should be independent of each
 - [Spearman Rank Correlation](https://statistics.laerd.com/statistical-guides/spearmans-rank-order-correlation-statistical-guide.php)
 
 
-## Seeking Signals - How to use `zipline`
+## From signals to trades: backtesting with `zipline`
 
 The open source [zipline](http://www.zipline.io/index.html) library is an event-driven backtesting system maintained and used in production by the crowd-sourced quantitative investment fund [Quantopian](https://www.quantopian.com/) to facilitate algorithm-development and live-trading. It automates the algorithm's reaction to trade events and provides it with current and historical point-in-time data that avoids look-ahead bias.
 
-- `zipline` installation: see [docs](http://www.zipline.io/index.html) and the introduction to `zipline` in [Chapter 2](../02_market_and_fundamental_data/02_data_providers/04_zipline) for more detail.
+### Installation
+
+- The current release 1.3 has a few shortcomings such as the [dependency on benchmark data from the IEX exchange](https://github.com/quantopian/zipline/issues/2480) and limitations for importing features beyond the basic OHLCV data points.
+- To enable the use of `zipline`, I've provided a [patched version](https://github.com/stefan-jansen/zipline) that works for the purposes of this book.
+    - Install by cloning the repo, `cd` into the packages' root folder and, after activating the `ml4t` environment, run `pip install -e`
 
 ## Separating signal and noise – how to use alphalens
 

04_alpha_factor_research/00_alpha_factors_in_practice/how_to_use_talib.ipynb

@@ -1112,7 +1112,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.8"
+   "version": "3.7.6"
   },
   "toc": {
    "base_numbering": 1,