{ "cells": [ { "cell_type": "markdown", "id": "eligible-asthma", "metadata": {}, "source": [ "# pyg.base.cell\n", "cell is a dict that forms part of a calculation graph. Most usefully, db_cell is implemented to maintain persistency of the function output in MongoDB. \n", "Before we start, we will show a few examples of how a cell works. Then, we will build a toy example of trading stocks based on an exponentially weighted crossover. \n" ] }, { "cell_type": "markdown", "id": "southwest-cedar", "metadata": {}, "source": [ "* We will start by creating the system using pyg.base.dictable and pyg.timeseries. \n", "* We then repeat the same code, this time modifying it slightly to save the data and calculation graph in MongoDB while running the calculation.\n", "* We conclude by discussing the two approaches" ] }, { "cell_type": "markdown", "id": "aboriginal-authorization", "metadata": {}, "source": [ "## Cell 101" ] }, { "cell_type": "code", "execution_count": 1, "id": "floppy-exchange", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "cell\n", "x:\n", " 2\n", "y:\n", " cell\n", " {'x': 1, 'y': 2, 'function': at 0x000002A68A888940>}\n", "function:\n", " at 0x000002A68A888700>" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from pyg import *\n", "a = cell(lambda x, y: x + y, x = 1, y = 2)\n", "b = cell(lambda x, y: x * y, x = 2, y = a)\n", "b" ] }, { "cell_type": "code", "execution_count": 2, "id": "normal-hollywood", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['x', 'y', 'function']" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "b.keys() ## b is a dict" ] }, { "cell_type": "code", "execution_count": 3, "id": "afraid-occasion", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['x', 'y']" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "b._args ## inputs" ] }, { "cell_type": "code", "execution_count": 4, "id": "tribal-enzyme", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['data']" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "b._output ## where the output will go once we calculate it" ] }, { "cell_type": "code", "execution_count": 5, "id": "numeric-cyprus", "metadata": {}, "outputs": [], "source": [ "assert b.run() ## b has not calculated yet... please run it" ] }, { "cell_type": "code", "execution_count": 6, "id": "personal-productivity", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "cell\n", "x:\n", " 2\n", "y:\n", " cell\n", " x:\n", " 1\n", " y:\n", " 2\n", " function:\n", " at 0x000002A68A888940>\n", " data:\n", " 3\n", "function:\n", " at 0x000002A68A888700>\n", "data:\n", " 6" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "b() # calculated object note b().data" ] }, { "cell_type": "code", "execution_count": 7, "id": "exceptional-packaging", "metadata": {}, "outputs": [], "source": [ "assert not b().run() ## b has calculated now... no need to run it" ] }, { "cell_type": "code", "execution_count": 8, "id": "judicial-organ", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "cell\n", "function:\n", " at 0x000002A68E0EF0D0>\n", "x:\n", " cell\n", " x:\n", " 1\n", " y:\n", " 2\n", " function:\n", " at 0x000002A68A888940>\n", " data:\n", " 3\n", "y:\n", " 2\n", "data:\n", " 9" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cell(lambda x, y: x ** y)(x = a, y = 2) # you can define the cell and then call it with the values" ] }, { "cell_type": "markdown", "id": "motivated-start", "metadata": {}, "source": [ "## Workflow without saving to the database" ] }, { "cell_type": "code", "execution_count": 9, "id": "registered-projector", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dictable[505 x 3]\n", "symbol|name |sector \n", "MMM |3M Company |Industrials\n", "AOS |A.O. Smith Corp |Industrials\n", "ABT |Abbott Laboratories |Health Care\n", "...505 rows...\n", "ZBH |Zimmer Biomet Holdings|Health Care\n", "ZION |Zions Bancorp |Financials \n", "ZTS |Zoetis |Health Care" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from pyg import *; \n", "import yfinance as yf # see https://github.com/ranaroussi/yfinance\n", "constituents = dictable(read_csv('d:/dropbox/yoav/python/pyg/docs/constituents_csv.csv')).rename(lower) # downloaded from \n", "constituents" ] }, { "cell_type": "code", "execution_count": 10, "id": "pending-fifty", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dictable[26 x 3]\n", "name |sector|symbol\n", "Apache Corporation|Energy|APA \n", "Baker Hughes Co |Energy|BKR \n", "Cabot Oil & Gas |Energy|COG \n", "...26 rows...\n", "TechnipFMC |Energy|FTI \n", "Valero Energy |Energy|VLO \n", "Williams Companies|Energy|WMB " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "stocks = constituents.inc(sector = 'Energy')\n", "stocks" ] }, { "cell_type": "code", "execution_count": 11, "id": "unable-festival", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "\n", "1 Failed download:\n", "- NBL: No data found, symbol may be delisted\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n", "[*********************100%***********************] 1 of 1 completed\n" ] } ], "source": [ "stocks = stocks(history = lambda symbol, sector, name: yf.download(tickers = symbol))" ] }, { "cell_type": "code", "execution_count": 12, "id": "written-blocking", "metadata": {}, "outputs": [], "source": [ "stocks = stocks.inc(lambda history: len(history)>0)" ] }, { "cell_type": "code", "execution_count": 13, "id": "formal-institution", "metadata": {}, "outputs": [], "source": [ "stocks = stocks(adj = lambda history: getitem(value = history, key = 'Adj Close'))" ] }, { "cell_type": "code", "execution_count": 14, "id": "respective-documentation", "metadata": {}, "outputs": [], "source": [ "stocks = stocks(rtn = lambda adj: diff(a = adj))" ] }, { "cell_type": "code", "execution_count": 15, "id": "south-kitty", "metadata": {}, "outputs": [], "source": [ "stocks = stocks(vol = lambda rtn: ewmstd(a = rtn, n = 30))" ] }, { "cell_type": "code", "execution_count": 33, "id": "duplicate-exchange", "metadata": {}, "outputs": [], "source": [ "_data = 'data'\n", "def crossover_(a, fast, slow, vol, instate = None):\n", " state = Dict(fast = {}, slow = {}, vol = {}) if instate is None else instate\n", " fast_ewma_ = ewma_(a, fast, instate = state.fast)\n", " slow_ewma_ = ewma_(a, slow, instate = state.slow) \n", " raw_signal = fast_ewma_.data - slow_ewma_.data\n", " signal_rms = ewmrms_(raw_signal, vol, instate = state.vol)\n", " normalized = raw_signal/v2na(signal_rms.data)\n", " return Dict(data = normalized, state = Dict(fast = fast_ewma_.state, slow = slow_ewma_.state, vol = signal_rms.state))\n", "\n", "crossover_.output = ['data', 'state']\n", "\n", "def crossover(a, fast, slow, vol, state = None):\n", " return crossover_(a, fast, slow, vol, instate = state)" ] }, { "cell_type": "markdown", "id": "korean-settlement", "metadata": {}, "source": [ "### some more functions to calculate the profits & loss as well as the signal/noise ratio" ] }, { "cell_type": "code", "execution_count": 17, "id": "second-factor", "metadata": {}, "outputs": [], "source": [ "def signal_pnl(signal, rtn, vol):\n", " return shift(signal) * (rtn/vol)\n", "\n", "def information_ratio(pnl):\n", " return 16 * ts_mean(pnl) / ts_std(pnl)" ] }, { "cell_type": "code", "execution_count": 18, "id": "separate-green", "metadata": {}, "outputs": [], "source": [ "forecasts = stocks * dictable(fast = [2,4,8], slow = [6,12,24], forecast = ['fast', 'medium', 'slow'])" ] }, { "cell_type": "code", "execution_count": 19, "id": "average-jewel", "metadata": {}, "outputs": [], "source": [ "forecasts = forecasts(signal = lambda rtn, fast, slow: crossover_(rtn, fast = fast, slow = slow, vol = 30).data)" ] }, { "cell_type": "code", "execution_count": 20, "id": "split-alexandria", "metadata": {}, "outputs": [], "source": [ "forecasts = forecasts(pnl = lambda signal, rtn, vol: signal_pnl(signal = signal, rtn = rtn, vol = vol))" ] }, { "cell_type": "code", "execution_count": 21, "id": "complex-characterization", "metadata": {}, "outputs": [], "source": [ "forecasts = forecasts(ir = lambda pnl: information_ratio(pnl = pnl))" ] }, { "cell_type": "code", "execution_count": 22, "id": "little-typing", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "symbol|fast |medium|slow \n", "APA |0.13 |-0.03 |-0.10\n", "BKR |0.06 |-0.10 |-0.14\n", "COG |0.20 |0.12 |-0.02\n", "COP |-0.14|-0.17 |-0.18\n", "CVX |0.47 |0.30 |0.11 \n", "CXO |0.01 |-0.14 |-0.34\n", "DVN |0.15 |0.15 |0.17 \n", "EOG |0.16 |0.05 |-0.03\n", "FANG |0.00 |-0.06 |-0.18\n", "FTI |-0.18|-0.37 |-0.37\n", "HAL |0.73 |0.47 |0.29 \n", "HES |0.16 |0.03 |0.00 \n", "HFC |0.86 |0.80 |0.71 \n", "KMI |0.45 |0.13 |0.03 \n", "MPC |0.21 |0.45 |0.72 \n", "MRO |0.24 |0.16 |0.14 \n", "NOV |0.10 |-0.04 |-0.04\n", "OKE |-0.22|-0.15 |-0.06\n", "OXY |-0.23|-0.29 |-0.22\n", "PSX |-0.02|0.11 |0.42 \n", "PXD |0.22 |0.17 |0.23 \n", "SLB |-0.08|-0.25 |-0.33\n", "VLO |0.30 |0.29 |0.41 \n", "WMB |0.17 |-0.08 |-0.22\n", "XOM |-0.03|-0.28 |-0.43\n" ] } ], "source": [ "print(forecasts.pivot('symbol', 'forecast', 'ir', [last, f12]))" ] }, { "cell_type": "markdown", "id": "mobile-labor", "metadata": {}, "source": [ "## Workflow while saving to MongoDB" ] }, { "cell_type": "markdown", "id": "loved-light", "metadata": {}, "source": [ "### Table creation\n", "We create three tables dependending on the primary keys we will be using. " ] }, { "cell_type": "code", "execution_count": 23, "id": "italian-mauritius", "metadata": {}, "outputs": [], "source": [ "idb = partial(mongo_table, db = 'demo', table = 'items', pk = 'item')\n", "sdb = partial(mongo_table, db = 'demo', table = 'stock', pk = ['item', 'symbol'])\n", "fdb = partial(mongo_table, db = 'demo', table = 'forecast', pk = ['item', 'symbol', 'forecast'])" ] }, { "cell_type": "code", "execution_count": 24, "id": "neutral-figure", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "ObjectId('602a566073531499b3861809')" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "idb().insert_one(Dict(item = 'constituents', data = constituents))" ] }, { "cell_type": "markdown", "id": "willing-tractor", "metadata": {}, "source": [ "### Any code differences?\n", "Most of the code remains the same as above, except:" ] }, { "cell_type": "markdown", "id": "macro-cooperative", "metadata": {}, "source": [ "* We wrap it inside a periodic_cell so it is calculated daily\n", "* We add reference to where we want to store it in MongoDB by specifying the db as well as the primary keys of that table\n", "* To run the function, we need to call the cell. This: loads the cell from the database (if found), checking if it even needs running and if so, runs it." ] }, { "cell_type": "code", "execution_count": 25, "id": "nominated-printing", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:31,197 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'APA'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:34,627 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'BKR'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:35,356 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'COG'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:36,122 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'CVX'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:38,242 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'CXO'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:38,754 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'COP'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:39,444 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'DVN'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:40,591 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'FANG'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:40,978 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'EOG'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:41,557 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'XOM'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:44,056 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'HAL'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:45,237 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'HES'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:46,323 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'HFC'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:47,026 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'KMI'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:48,145 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'MRO'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:50,428 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'MPC'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:51,781 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'NOV'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:52,541 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'OXY'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:54,166 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'OKE'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:55,333 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'PSX'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:55,735 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'PXD'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:56,563 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'SLB'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:58,319 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'FTI'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:58,844 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'VLO'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 19:59:59,789 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'WMB'))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[*********************100%***********************] 1 of 1 completed\n" ] } ], "source": [ "stocks = stocks(history = lambda symbol, sector, name: periodic_cell(yf.download, tickers = symbol, # these are the inputs for the function\n", " db = sdb, item = 'history', symbol = symbol)()) # these define where the data goes to" ] }, { "cell_type": "markdown", "id": "artificial-twenty", "metadata": {}, "source": [ "### Accessing the data in MongoDB\n", "The data is now in the database and can be accessed:" ] }, { "cell_type": "code", "execution_count": 26, "id": "positive-being", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
OpenHighLowCloseAdj CloseVolume
Date
1970-01-026.8515636.8906256.8437506.8515631.47123272000
1970-01-056.8593756.8984386.8593756.8906251.479620446400
1970-01-066.8906256.9609386.8828136.9609381.494717176000
1970-01-076.9609387.0156256.9453137.0000001.503106164800
1970-01-087.0000007.1093756.9843757.0937501.523237304000
.....................
2021-02-08179.300003180.869995179.169998180.759995179.2826082355100
2021-02-09181.220001181.899994180.179993180.940002179.4611511942800
2021-02-10181.880005182.380005180.639999181.080002179.6000061929000
2021-02-11179.350006179.880005175.839996177.210007177.2100072187100
2021-02-12177.270004178.839996177.210007178.699997178.6999971081500
\n", "

12895 rows × 6 columns

\n", "
" ], "text/plain": [ " Open High Low Close Adj Close \\\n", "Date \n", "1970-01-02 6.851563 6.890625 6.843750 6.851563 1.471232 \n", "1970-01-05 6.859375 6.898438 6.859375 6.890625 1.479620 \n", "1970-01-06 6.890625 6.960938 6.882813 6.960938 1.494717 \n", "1970-01-07 6.960938 7.015625 6.945313 7.000000 1.503106 \n", "1970-01-08 7.000000 7.109375 6.984375 7.093750 1.523237 \n", "... ... ... ... ... ... \n", "2021-02-08 179.300003 180.869995 179.169998 180.759995 179.282608 \n", "2021-02-09 181.220001 181.899994 180.179993 180.940002 179.461151 \n", "2021-02-10 181.880005 182.380005 180.639999 181.080002 179.600006 \n", "2021-02-11 179.350006 179.880005 175.839996 177.210007 177.210007 \n", "2021-02-12 177.270004 178.839996 177.210007 178.699997 178.699997 \n", "\n", " Volume \n", "Date \n", "1970-01-02 72000 \n", "1970-01-05 446400 \n", "1970-01-06 176000 \n", "1970-01-07 164800 \n", "1970-01-08 304000 \n", "... ... \n", "2021-02-08 2355100 \n", "2021-02-09 1942800 \n", "2021-02-10 1929000 \n", "2021-02-11 2187100 \n", "2021-02-12 1081500 \n", "\n", "[12895 rows x 6 columns]" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "get_data('stock', 'demo', symbol = 'MMM')" ] }, { "cell_type": "code", "execution_count": 27, "id": "female-technician", "metadata": {}, "outputs": [], "source": [ "stocks = stocks.inc(lambda history: len(history.data)>0)" ] }, { "cell_type": "code", "execution_count": 28, "id": "knowing-herald", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 20:00:02,042 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'APA'))\n", "2021-03-06 20:00:03,128 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'BKR'))\n", "2021-03-06 20:00:03,468 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'COG'))\n", "2021-03-06 20:00:03,773 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'CVX'))\n", "2021-03-06 20:00:04,094 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'CXO'))\n", "2021-03-06 20:00:04,340 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'COP'))\n", "2021-03-06 20:00:04,633 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'DVN'))\n", "2021-03-06 20:00:04,936 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'FANG'))\n", "2021-03-06 20:00:05,160 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'EOG'))\n", "2021-03-06 20:00:05,518 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'XOM'))\n", "2021-03-06 20:00:05,860 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'HAL'))\n", "2021-03-06 20:00:06,272 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'HES'))\n", "2021-03-06 20:00:06,881 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'HFC'))\n", "2021-03-06 20:00:07,289 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'KMI'))\n", "2021-03-06 20:00:07,601 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'MRO'))\n", "2021-03-06 20:00:07,905 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'MPC'))\n", "2021-03-06 20:00:08,608 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'NOV'))\n", "2021-03-06 20:00:09,233 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'OXY'))\n", "2021-03-06 20:00:09,762 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'OKE'))\n", "2021-03-06 20:00:10,238 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'PSX'))\n", "2021-03-06 20:00:11,431 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'PXD'))\n", "2021-03-06 20:00:13,090 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'SLB'))\n", "2021-03-06 20:00:14,979 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'FTI'))\n", "2021-03-06 20:00:15,767 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'VLO'))\n", "2021-03-06 20:00:16,145 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'WMB'))\n" ] } ], "source": [ "stocks = stocks(adj = lambda history, symbol: periodic_cell(getitem, value = history, key = 'Adj Close', \n", " db = sdb, symbol = symbol, item = 'adj')()) " ] }, { "cell_type": "code", "execution_count": 29, "id": "portable-allowance", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 20:00:16,480 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'APA'))\n", "2021-03-06 20:00:16,930 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'BKR'))\n", "2021-03-06 20:00:17,348 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'COG'))\n", "2021-03-06 20:00:17,656 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'CVX'))\n", "2021-03-06 20:00:18,515 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'CXO'))\n", "2021-03-06 20:00:18,816 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'COP'))\n", "2021-03-06 20:00:19,750 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'DVN'))\n", "2021-03-06 20:00:20,084 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'FANG'))\n", "2021-03-06 20:00:20,322 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'EOG'))\n", "2021-03-06 20:00:20,813 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'XOM'))\n", "2021-03-06 20:00:21,168 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'HAL'))\n", "2021-03-06 20:00:21,636 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'HES'))\n", "2021-03-06 20:00:22,050 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'HFC'))\n", "2021-03-06 20:00:22,342 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'KMI'))\n", "2021-03-06 20:00:22,764 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'MRO'))\n", "2021-03-06 20:00:23,127 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'MPC'))\n", "2021-03-06 20:00:23,383 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'NOV'))\n", "2021-03-06 20:00:23,681 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'OXY'))\n", "2021-03-06 20:00:23,978 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'OKE'))\n", "2021-03-06 20:00:24,327 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'PSX'))\n", "2021-03-06 20:00:24,580 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'PXD'))\n", "2021-03-06 20:00:24,850 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'SLB'))\n", "2021-03-06 20:00:25,313 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'FTI'))\n", "2021-03-06 20:00:25,611 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'VLO'))\n", "2021-03-06 20:00:25,938 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'WMB'))\n" ] } ], "source": [ "stocks = stocks(rtn = lambda adj, symbol: periodic_cell(diff, a = adj, \n", " db = sdb, symbol = symbol, item = 'rtn')())" ] }, { "cell_type": "code", "execution_count": 30, "id": "urban-machine", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 20:00:26,300 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'APA'))\n", "2021-03-06 20:00:26,595 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'BKR'))\n", "2021-03-06 20:00:26,844 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'COG'))\n", "2021-03-06 20:00:27,498 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'CVX'))\n", "2021-03-06 20:00:28,895 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'CXO'))\n", "2021-03-06 20:00:29,513 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'COP'))\n", "2021-03-06 20:00:30,345 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'DVN'))\n", "2021-03-06 20:00:31,002 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'FANG'))\n", "2021-03-06 20:00:31,378 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'EOG'))\n", "2021-03-06 20:00:31,711 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'XOM'))\n", "2021-03-06 20:00:34,153 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'HAL'))\n", "2021-03-06 20:00:35,635 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'HES'))\n", "2021-03-06 20:00:36,165 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'HFC'))\n", "2021-03-06 20:00:36,492 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'KMI'))\n", "2021-03-06 20:00:37,163 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'MRO'))\n", "2021-03-06 20:00:37,823 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'MPC'))\n", "2021-03-06 20:00:38,481 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'NOV'))\n", "2021-03-06 20:00:39,119 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'OXY'))\n", "2021-03-06 20:00:39,425 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'OKE'))\n", "2021-03-06 20:00:39,705 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'PSX'))\n", "2021-03-06 20:00:40,127 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'PXD'))\n", "2021-03-06 20:00:40,378 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'SLB'))\n", "2021-03-06 20:00:40,631 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'FTI'))\n", "2021-03-06 20:00:41,056 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'VLO'))\n", "2021-03-06 20:00:41,304 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'WMB'))\n" ] } ], "source": [ "stocks = stocks(vol = lambda rtn, symbol: periodic_cell(ewmstd, a = rtn, n = 30, \n", " db = sdb, symbol = symbol, item = 'vol')())" ] }, { "cell_type": "markdown", "id": "mighty-friendship", "metadata": {}, "source": [ "### Calculating the forecasts & saving them" ] }, { "cell_type": "code", "execution_count": 31, "id": "editorial-member", "metadata": {}, "outputs": [], "source": [ "forecasts = stocks * dictable(fast = [2,4,8], slow = [6,12,24], forecast = ['fast', 'medium', 'slow'])" ] }, { "cell_type": "code", "execution_count": 34, "id": "competitive-settlement", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 20:04:15,290 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'APA'))\n", "2021-03-06 20:04:52,869 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'APA'))\n", "2021-03-06 20:04:53,186 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'APA'))\n", "2021-03-06 20:04:53,434 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'BKR'))\n", "2021-03-06 20:04:53,681 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'BKR'))\n", "2021-03-06 20:04:53,913 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'BKR'))\n", "2021-03-06 20:04:54,203 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'COG'))\n", "2021-03-06 20:04:56,089 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'COG'))\n", "2021-03-06 20:04:56,727 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'COG'))\n", "2021-03-06 20:04:57,250 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'CVX'))\n", "2021-03-06 20:04:58,763 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'CVX'))\n", "2021-03-06 20:04:59,116 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'CVX'))\n", "2021-03-06 20:04:59,518 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'CXO'))\n", "2021-03-06 20:05:00,010 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'CXO'))\n", "2021-03-06 20:05:00,371 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'CXO'))\n", "2021-03-06 20:05:00,771 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'COP'))\n", "2021-03-06 20:05:01,088 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'COP'))\n", "2021-03-06 20:05:01,374 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'COP'))\n", "2021-03-06 20:05:01,673 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'DVN'))\n", "2021-03-06 20:05:01,948 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'DVN'))\n", "2021-03-06 20:05:02,256 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'DVN'))\n", "2021-03-06 20:05:02,500 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'FANG'))\n", "2021-03-06 20:05:02,708 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'FANG'))\n", "2021-03-06 20:05:02,927 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'FANG'))\n", "2021-03-06 20:05:03,134 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'EOG'))\n", "2021-03-06 20:05:03,417 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'EOG'))\n", "2021-03-06 20:05:03,671 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'EOG'))\n", "2021-03-06 20:05:03,898 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'XOM'))\n", "2021-03-06 20:05:04,224 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'XOM'))\n", "2021-03-06 20:05:04,485 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'XOM'))\n", "2021-03-06 20:05:04,701 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'HAL'))\n", "2021-03-06 20:05:05,394 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'HAL'))\n", "2021-03-06 20:05:05,652 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'HAL'))\n", "2021-03-06 20:05:05,902 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'HES'))\n", "2021-03-06 20:05:06,130 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'HES'))\n", "2021-03-06 20:05:06,390 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'HES'))\n", "2021-03-06 20:05:06,616 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'HFC'))\n", "2021-03-06 20:05:06,883 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'HFC'))\n", "2021-03-06 20:05:07,099 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'HFC'))\n", "2021-03-06 20:05:07,321 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'KMI'))\n", "2021-03-06 20:05:07,512 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'KMI'))\n", "2021-03-06 20:05:07,718 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'KMI'))\n", "2021-03-06 20:05:07,941 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'MRO'))\n", "2021-03-06 20:05:08,197 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'MRO'))\n", "2021-03-06 20:05:08,458 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'MRO'))\n", "2021-03-06 20:05:08,687 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'MPC'))\n", "2021-03-06 20:05:08,905 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'MPC'))\n", "2021-03-06 20:05:09,129 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'MPC'))\n", "2021-03-06 20:05:09,348 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'NOV'))\n", "2021-03-06 20:05:09,620 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'NOV'))\n", "2021-03-06 20:05:09,845 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'NOV'))\n", "2021-03-06 20:05:10,052 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'OXY'))\n", "2021-03-06 20:05:10,383 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'OXY'))\n", "2021-03-06 20:05:10,705 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'OXY'))\n", "2021-03-06 20:05:10,958 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'OKE'))\n", "2021-03-06 20:05:11,232 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'OKE'))\n", "2021-03-06 20:05:11,475 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'OKE'))\n", "2021-03-06 20:05:11,715 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'PSX'))\n", "2021-03-06 20:05:11,997 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'PSX'))\n", "2021-03-06 20:05:12,207 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'PSX'))\n", "2021-03-06 20:05:12,414 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'PXD'))\n", "2021-03-06 20:05:12,669 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'PXD'))\n", "2021-03-06 20:05:12,901 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'PXD'))\n", "2021-03-06 20:05:13,128 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'SLB'))\n", "2021-03-06 20:05:13,751 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'SLB'))\n", "2021-03-06 20:05:13,985 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'SLB'))\n", "2021-03-06 20:05:14,230 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'FTI'))\n", "2021-03-06 20:05:14,503 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'FTI'))\n", "2021-03-06 20:05:14,775 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'FTI'))\n", "2021-03-06 20:05:15,037 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'VLO'))\n", "2021-03-06 20:05:15,346 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'VLO'))\n", "2021-03-06 20:05:15,578 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'VLO'))\n", "2021-03-06 20:05:15,814 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'WMB'))\n", "2021-03-06 20:05:16,067 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'WMB'))\n", "2021-03-06 20:05:16,283 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'WMB'))\n" ] } ], "source": [ "forecasts = forecasts(signal = lambda rtn, fast, slow, symbol, forecast: periodic_cell(crossover_, a = rtn, fast = fast, slow = slow, vol = 30,\n", " db = fdb, symbol = symbol, forecast = forecast, item = 'signal')())" ] }, { "cell_type": "code", "execution_count": 36, "id": "changed-compression", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-06 20:07:41,567 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'APA'))\n", "2021-03-06 20:09:01,609 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'APA'))\n", "2021-03-06 20:09:02,368 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'APA'))\n", "2021-03-06 20:09:03,885 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'BKR'))\n", "2021-03-06 20:09:04,804 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'BKR'))\n", "2021-03-06 20:09:05,484 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'BKR'))\n", "2021-03-06 20:09:06,030 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'COG'))\n", "2021-03-06 20:09:06,627 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'COG'))\n", "2021-03-06 20:09:07,453 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'COG'))\n", "2021-03-06 20:09:08,153 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'CVX'))\n", "2021-03-06 20:09:09,125 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'CVX'))\n", "2021-03-06 20:09:09,730 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'CVX'))\n", "2021-03-06 20:09:10,582 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'CXO'))\n", "2021-03-06 20:09:11,216 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'CXO'))\n", "2021-03-06 20:09:11,904 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'CXO'))\n", "2021-03-06 20:09:12,350 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'COP'))\n", "2021-03-06 20:09:13,415 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'COP'))\n", "2021-03-06 20:09:14,255 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'COP'))\n", "2021-03-06 20:09:15,747 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'DVN'))\n", "2021-03-06 20:09:16,408 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'DVN'))\n", "2021-03-06 20:09:17,055 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'DVN'))\n", "2021-03-06 20:09:17,802 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'FANG'))\n", "2021-03-06 20:09:18,207 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'FANG'))\n", "2021-03-06 20:09:18,601 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'FANG'))\n", "2021-03-06 20:09:20,074 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'EOG'))\n", "2021-03-06 20:09:20,820 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'EOG'))\n", "2021-03-06 20:09:21,605 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'EOG'))\n", "2021-03-06 20:09:23,377 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'XOM'))\n", "2021-03-06 20:09:25,407 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'XOM'))\n", "2021-03-06 20:09:27,902 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'XOM'))\n", "2021-03-06 20:09:29,626 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'HAL'))\n", "2021-03-06 20:09:31,617 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'HAL'))\n", "2021-03-06 20:09:32,908 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'HAL'))\n", "2021-03-06 20:09:34,001 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'HES'))\n", "2021-03-06 20:09:35,216 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'HES'))\n", "2021-03-06 20:09:35,767 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'HES'))\n", "2021-03-06 20:09:36,840 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'HFC'))\n", "2021-03-06 20:09:38,454 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'HFC'))\n", "2021-03-06 20:09:38,896 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'HFC'))\n", "2021-03-06 20:09:39,445 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'KMI'))\n", "2021-03-06 20:09:40,384 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'KMI'))\n", "2021-03-06 20:09:40,983 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'KMI'))\n", "2021-03-06 20:09:41,528 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'MRO'))\n", "2021-03-06 20:09:43,164 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'MRO'))\n", "2021-03-06 20:09:44,007 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'MRO'))\n", "2021-03-06 20:09:44,933 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'MPC'))\n", "2021-03-06 20:09:45,778 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'MPC'))\n", "2021-03-06 20:09:46,376 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'MPC'))\n", "2021-03-06 20:09:46,926 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'NOV'))\n", "2021-03-06 20:09:47,990 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'NOV'))\n", "2021-03-06 20:09:48,697 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'NOV'))\n", "2021-03-06 20:09:49,380 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'OXY'))\n", "2021-03-06 20:09:50,123 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'OXY'))\n", "2021-03-06 20:09:51,039 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'OXY'))\n", "2021-03-06 20:09:52,698 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'OKE'))\n", "2021-03-06 20:09:54,721 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'OKE'))\n", "2021-03-06 20:09:55,441 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'OKE'))\n", "2021-03-06 20:09:56,279 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'PSX'))\n", "2021-03-06 20:09:56,974 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'PSX'))\n", "2021-03-06 20:09:57,510 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'PSX'))\n", "2021-03-06 20:09:57,980 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'PXD'))\n", "2021-03-06 20:09:58,568 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'PXD'))\n", "2021-03-06 20:09:59,271 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'PXD'))\n", "2021-03-06 20:10:00,220 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'SLB'))\n", "2021-03-06 20:10:02,318 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'SLB'))\n", "2021-03-06 20:10:03,380 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'SLB'))\n", "2021-03-06 20:10:04,161 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'FTI'))\n", "2021-03-06 20:10:04,744 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'FTI'))\n", "2021-03-06 20:10:05,535 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'FTI'))\n", "2021-03-06 20:10:06,195 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'VLO'))\n", "2021-03-06 20:10:07,851 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'VLO'))\n", "2021-03-06 20:10:09,025 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'VLO'))\n", "2021-03-06 20:10:10,054 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'WMB'))\n", "2021-03-06 20:10:11,128 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'WMB'))\n", "2021-03-06 20:10:11,790 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'WMB'))\n" ] } ], "source": [ "forecasts = forecasts(pnl = lambda signal, rtn, vol, symbol, forecast: periodic_cell(signal_pnl, signal = signal, rtn = rtn, vol = vol,\n", " db = fdb, symbol = symbol, forecast = forecast, item = 'pnl')())" ] }, { "cell_type": "code", "execution_count": 37, "id": "advisory-samuel", "metadata": {}, "outputs": [], "source": [ "forecasts = forecasts(ir = lambda pnl: information_ratio(pnl = pnl.data))" ] }, { "cell_type": "code", "execution_count": 38, "id": "constitutional-banner", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "symbol|fast |medium|slow \n", "APA |0.13 |-0.03 |-0.10\n", "BKR |0.06 |-0.10 |-0.14\n", "COG |0.20 |0.12 |-0.02\n", "COP |-0.14|-0.17 |-0.18\n", "CVX |0.47 |0.30 |0.11 \n", "CXO |0.01 |-0.14 |-0.34\n", "DVN |0.15 |0.15 |0.17 \n", "EOG |0.16 |0.05 |-0.03\n", "FANG |0.00 |-0.06 |-0.18\n", "FTI |-0.18|-0.37 |-0.37\n", "HAL |0.73 |0.47 |0.29 \n", "HES |0.16 |0.03 |0.00 \n", "HFC |0.86 |0.80 |0.71 \n", "KMI |0.45 |0.13 |0.03 \n", "MPC |0.21 |0.45 |0.72 \n", "MRO |0.24 |0.16 |0.14 \n", "NOV |0.10 |-0.04 |-0.04\n", "OKE |-0.22|-0.15 |-0.06\n", "OXY |-0.23|-0.29 |-0.22\n", "PSX |-0.02|0.11 |0.42 \n", "PXD |0.22 |0.17 |0.23 \n", "SLB |-0.08|-0.25 |-0.33\n", "VLO |0.30 |0.29 |0.41 \n", "WMB |0.17 |-0.08 |-0.22\n", "XOM |-0.03|-0.28 |-0.43\n" ] } ], "source": [ "print(forecasts.pivot('symbol', 'forecast', 'ir', [last, f12]))" ] }, { "cell_type": "markdown", "id": "historic-score", "metadata": {}, "source": [ "## Accessing & running the graph once the graph has been created\n", "We can access the data or the cell:" ] }, { "cell_type": "code", "execution_count": 39, "id": "specific-layer", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "periodic_cell\n", "updated:\n", " 2021-03-06 20:04:51.410000\n", "_period:\n", " 1b\n", "db:\n", " functools.partial(, db='demo', table='forecast', pk=['item', 'symbol', 'forecast'])\n", "_id:\n", " 602a86fb4de6ffaf1c045c8f\n", "_pk:\n", " ['forecast', 'item', 'symbol']\n", "a:\n", " periodic_cell\n", " updated:\n", " None\n", " _period:\n", " 1b\n", " db:\n", " functools.partial(, db='demo', table='stock', pk=['item', 'symbol'])\n", " item:\n", " rtn\n", " symbol:\n", " APA\n", " function:\n", " None\n", "data:\n", " Date\n", " 1979-05-15 NaN\n", " 1979-05-16 NaN\n", " 1979-05-17 -1.402762\n", " 1979-05-18 -0.939334\n", " 1979-05-21 -1.194304\n", " ... \n", " 2021-03-01 -1.018576\n", " 2021-03-02 -0.831244\n", " 2021-03-03 -0.187666\n", " 2021-03-04 0.972348\n", " 2021-03-05 2.674065\n", " Length: 10543, dtype: float64\n", "fast:\n", " 2\n", "forecast:\n", " fast\n", "instate:\n", " None\n", "item:\n", " signal\n", "slow:\n", " 6\n", "state:\n", " Dict\n", " fast:\n", " Dict\n", " state:\n", " {'t': nan, 't0': 0.9999999999999999, 't1': 0.46353936200681184}\n", " t:\n", " nan\n", " t0:\n", " 0.9999999999999999\n", " t1:\n", " 1.0413438705045697\n", " slow:\n", " Dict\n", " state:\n", " {'t': nan, 't0': 0.9999999999999997, 't1': 0.2672723621042594}\n", " t:\n", " nan\n", " t0:\n", " 0.9999999999999997\n", " t1:\n", " 0.5552029895351339\n", " vol:\n", " Dict\n", " state:\n", " {'t': nan, 't0': 0.9999999999999983, 't2': 0.02845240938173902}\n", " t:\n", " nan\n", " t0:\n", " 0.9999999999999983\n", " t2:\n", " 0.033050680992027265\n", "symbol:\n", " APA\n", "vol:\n", " 30\n", "function:\n", " " ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "get_cell('forecast', 'demo', symbol = 'APA', forecast = 'fast', item = 'signal')" ] }, { "cell_type": "markdown", "id": "occasional-occasions", "metadata": {}, "source": [ "And now that the graph has been created, you can actually trigger it just by loading. i.e. The code below will give you the fast signal for APA and will ensure it is up-to-date too:" ] }, { "cell_type": "code", "execution_count": 40, "id": "irish-chicken", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Date\n", "1979-05-15 NaN\n", "1979-05-16 NaN\n", "1979-05-17 -1.402762\n", "1979-05-18 -0.939334\n", "1979-05-21 -1.194304\n", " ... \n", "2021-03-01 -1.018576\n", "2021-03-02 -0.831244\n", "2021-03-03 -0.187666\n", "2021-03-04 0.972348\n", "2021-03-05 2.674065\n", "Length: 10543, dtype: float64\n" ] } ], "source": [ "c = get_cell('forecast', 'demo', symbol = 'APA', forecast = 'fast', item = 'signal')\n", "c = c.go()\n", "print(c.data)" ] }, { "cell_type": "markdown", "id": "million-google", "metadata": {}, "source": [ "## Point-in-time, cache and persistency\n", "The pk-tables save a full history of all your data. \n", "To avoid hitting the database all the time, we also have a local GRAPH singleton that caches all the cells by their address. The cell has few basic operations we need to understand:" ] }, { "cell_type": "markdown", "id": "informational-tutorial", "metadata": {}, "source": [ "* **cell.run()**: Returns True/False if the cell needs to be calculated. db_cell() just check for values in its output, periodic_cell will also check if it is a new business day.\n", "* **cell.go()**: This calculates the cell and saves the result to the database (and to GRAPH). The cell itself is not loaded but all its inputs are loaded\n", " - cell.go(0) : calculate only if there is a need\n", " - cell.go(1) : calculate me but my parents only if there is a need\n", " - cell.go(2) : calculate me & my parents but my grandparents only if there is a need\n", " - cell.go(-1) : calculate everything\n", "
\n", "* **cell.load()**: This loads the data from GRAPH, if not in GRAPH, loads it from MongoDB (and updates also the GRAPH) \n", " - cell.load(-1) : Clear the data from the GRAPH\n", " - cell.load(0) : Load & update me from GRAPH, if not, from MongoDB, if not, just return good old me\n", " - cell.load(1) : If you cannot find me, throw an Exception\n", " - cell.load(date) : Load my version as valid on date. If none exists, throw.\n", " - cell.load([value]): Force GRAPH to clear, only load from DB. Same as cell.load(-1).load(value) \n", "* **cell()**: This loads & then go" ] }, { "cell_type": "code", "execution_count": 51, "id": "australian-clearing", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-19 08:42:10,778 - pyg - INFO - INFO: deleting 23 documents based on M{}\n" ] } ], "source": [ "from pyg import *; from functools import partial\n", "db = partial(mongo_table, db = 'demo', table = 'persistency', pk = 'key')\n", "db().raw.drop()\n", "\n", "def f(a, b):\n", " return a+b" ] }, { "cell_type": "code", "execution_count": 56, "id": "continent-shock", "metadata": {}, "outputs": [], "source": [ "## now we set up a fake calculation tree:\n", "x = db_cell(f, a = 1, b = 2, db = db, key = 'x')\n", "y = db_cell(f, a = x, b = 2, db = db, key = 'y')\n", "z = db_cell(f, a = x, b = y, db = db, key = 'z')\n", "\n", "## and run it by running the final value we want\n", "z = z()" ] }, { "cell_type": "code", "execution_count": 57, "id": "sunset-absence", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "3" ] }, "execution_count": 57, "metadata": {}, "output_type": "execute_result" } ], "source": [ "## we can access the data:\n", "get_data('persistency', 'demo', key = 'x')" ] }, { "cell_type": "code", "execution_count": 58, "id": "realistic-williams", "metadata": {}, "outputs": [], "source": [ "t0 = dt() ## first breakpoint" ] }, { "cell_type": "code", "execution_count": 60, "id": "annoying-essence", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-19 08:43:07,223 - pyg - INFO - ('localhost', 27017, 'demo', 'persistency', ('key',), ('x',))\n", "2021-03-19 08:43:07,317 - pyg - INFO - ('localhost', 27017, 'demo', 'persistency', ('key',), ('y',))\n", "2021-03-19 08:43:07,547 - pyg - INFO - ('localhost', 27017, 'demo', 'persistency', ('key',), ('z',))\n" ] } ], "source": [ "x = db_cell(f, a = 10, b = 20, db = db, key = 'x').go()\n", "y = db_cell(f, a = x, b = 20, db = db, key = 'y').go()\n", "z = db_cell(f, a = x, b = y, db = db, key = 'z').go()" ] }, { "cell_type": "code", "execution_count": 61, "id": "extreme-string", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "30" ] }, "execution_count": 61, "metadata": {}, "output_type": "execute_result" } ], "source": [ "## and here is the new data\n", "get_data('persistency', 'demo', key = 'x')" ] }, { "cell_type": "code", "execution_count": 62, "id": "another-hours", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "3" ] }, "execution_count": 62, "metadata": {}, "output_type": "execute_result" } ], "source": [ "## and here is the data valid at our first breakpoint. get_data/get_cell always go to the database to load values\n", "get_data('persistency', 'demo', key = 'x', _deleted = t0)" ] }, { "cell_type": "markdown", "id": "designed-reasoning", "metadata": {}, "source": [ "We can ask a cell to load itself, but remember: it will go to GRAPH first by default. The GRAPH has only one copy of the cell, while in MongoDB, every time we recalculate/save a new version of the cell, we mark the old version in the database as \"deleted\" but otherwise keep it. To force a cell to load itself from the database, encase the breakpoint in a list..." ] }, { "cell_type": "code", "execution_count": 63, "id": "mysterious-marks", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "db_cell\n", "db:\n", " functools.partial(, db='demo', table='persistency', pk='key')\n", "_id:\n", " 60546402e49a0e77dca6dff3\n", "_pk:\n", " ['key']\n", "a:\n", " 1\n", "b:\n", " 2\n", "data:\n", " 3\n", "key:\n", " x\n", "_deleted:\n", " 2021-03-19 08:42:42.668000\n", "function:\n", " " ] }, "execution_count": 63, "metadata": {}, "output_type": "execute_result" } ], "source": [ "db_cell(db = db, key = 'x').load([t0])" ] }, { "cell_type": "markdown", "id": "armed-boston", "metadata": {}, "source": [ "We can force a full recalculation of the tree in a single line of code:" ] }, { "cell_type": "code", "execution_count": 82, "id": "julian-orbit", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-19 08:58:39,317 - pyg - INFO - ('localhost', 27017, 'demo', 'persistency', ('key',), ('z',))\n", "2021-03-19 08:58:39,946 - pyg - INFO - ('localhost', 27017, 'demo', 'persistency', ('key',), ('x',))\n", "2021-03-19 08:58:40,087 - pyg - INFO - ('localhost', 27017, 'demo', 'persistency', ('key',), ('y',))\n", "2021-03-19 08:58:40,213 - pyg - INFO - ('localhost', 27017, 'demo', 'persistency', ('key',), ('x',))\n" ] }, { "data": { "text/plain": [ "8" ] }, "execution_count": 82, "metadata": {}, "output_type": "execute_result" } ], "source": [ "db_cell(db = db, key = 'z')(go = -1, mode = [t0]).data ## Should be 8, same as the old value" ] }, { "cell_type": "markdown", "id": "varied-gilbert", "metadata": {}, "source": [ "## Comparison of the two workflows\n", "Saving to the database has negatives:" ] }, { "cell_type": "markdown", "id": "secret-peripheral", "metadata": {}, "source": [ "- does require some (but really not much) additional code to specify where each data item goes to\n", "- slows down the calculation" ] }, { "cell_type": "markdown", "id": "sunset-transmission", "metadata": {}, "source": [ "Conversely, " ] }, { "cell_type": "markdown", "id": "mexican-webster", "metadata": {}, "source": [ "+ We get full persistency: We can access each part of the graph with full visibility on the inputs, the function used to calculate the result, the function output(s), the location of where the data is stored and the time it was last updated as well as the periodicity it is calculated.\n", "+ We get full audit, past calculations remain available to track (and indeed, rerun) if anything goes wrong\n", "+ Each node will manage its schedule, ensuring data is up-to-date \n", "+ We can run just the parts of the graph we are interested in (and can run in parallel)" ] }, { "cell_type": "markdown", "id": "armed-husband", "metadata": {}, "source": [ "## To save or not to save?\n", "Luckily we don't really need to decide on one workflow or the other as both can happily coexist. \n", "
We can build a calculation graph and decide that some key points in the calculation we want to save while intermediate calculations we can calculate on the fly and not save at all. \n", "
We have met the crossover function. Here we implement it 'on the fly' while saving just final value to db" ] }, { "cell_type": "code", "execution_count": 1, "id": "arabic-hardware", "metadata": {}, "outputs": [], "source": [ "from pyg import *; import pandas as pd; import numpy as np; from functools import partial" ] }, { "cell_type": "code", "execution_count": 2, "id": "civic-calculation", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-14 00:08:03,885 - pyg - INFO - INFO: deleting 7 documents based on M{}\n" ] }, { "data": { "text/plain": [ " for Collection(Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True), 'test'), 'test') \n", "M{} None\n", "documents count: 0" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def fake_ts(ticker):\n", " return pd.Series(np.random.normal(0,1,1000), drange(-999))\n", "db = partial(mongo_table, db = 'test', table = 'test', pk = ['key'])\n", "db().raw.drop()" ] }, { "cell_type": "code", "execution_count": 3, "id": "adjusted-consumption", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-14 00:08:04,243 - pyg - INFO - ('localhost', 27017, 'test', 'test', ('key',), ('appl_rtn',))\n", "2021-03-14 00:08:04,416 - pyg - INFO - ('localhost', 27017, 'test', 'test', ('key',), ('appl_crossover',))\n", "2021-03-14 00:08:04,419 - pyg - INFO - (None, None, None, None, ('key',), ('calculate difference of ewma',))\n", "2021-03-14 00:08:05,162 - pyg - INFO - (None, None, None, None, ('key',), ('root mean square of difference',))\n", "2021-03-14 00:08:05,164 - pyg - INFO - (None, None, None, None, ('key',), ('calculate difference of ewma',))\n" ] } ], "source": [ "appl = db_cell(fake_ts, ticker = 'appl', key = 'appl_rtn', db = db)()\n", "\n", "#I am never saving these, In fact, I don't want to see these in calculation log.\n", "a = cell(ewma, a = appl, n = 30) \n", "b = cell(ewma, a = appl, n = 50)\n", "\n", "# I may want to save these nodes but haven't made up my mind\n", "# I do want to see the calculations in the log though...\n", "# I replace db by the primary keys of table (here 'key'). \n", "# This allows as to see the calculation log as it happens. \n", "# data is not saved to db though until I switch to db = db as opposed to db = 'key'\n", "\n", "c = db_cell(sub_, a = a, b = b, key = 'calculate difference of ewma', db = 'key') \n", "d = db_cell(ewmrms, a = c, n = 100, key = 'root mean square of difference', db = 'key')\n", "\n", "# The final crossover I definitely want to save to db: \n", "final_value = db_cell(div_, a = c, b = d, key = 'appl_crossover', db = db)()" ] }, { "cell_type": "code", "execution_count": 4, "id": "former-salem", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['appl_crossover', 'appl_rtn']" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "db().key" ] }, { "cell_type": "markdown", "id": "floppy-routine", "metadata": {}, "source": [ "So although we had several intermediate steps, we decided to save just the final crossover in the database, If we look at the inputs for the function, you can see that the values are not saved in the database, though the full calculation tree _is_. \n", "
Therefore, one can reload the node and then recalculate all the intermediate values on the fly" ] }, { "cell_type": "code", "execution_count": 5, "id": "another-architect", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-03-14 00:08:09,934 - pyg - INFO - ('localhost', 27017, 'test', 'test', ('key',), ('appl_crossover',))\n", "2021-03-14 00:08:09,936 - pyg - INFO - (None, None, None, None, ('key',), ('calculate difference of ewma',))\n", "2021-03-14 00:08:10,091 - pyg - INFO - (None, None, None, None, ('key',), ('root mean square of difference',))\n", "2021-03-14 00:08:10,095 - pyg - INFO - (None, None, None, None, ('key',), ('calculate difference of ewma',))\n" ] } ], "source": [ "loaded_and_recalculated = (db()[dict(key = 'appl_crossover')] - 'data').go(1) ## but once recalculated, we can assert we got the same result\n", "assert eq(loaded_and_recalculated.data, final_value.data)" ] }, { "cell_type": "markdown", "id": "jewish-corruption", "metadata": {}, "source": [ "## Behind the scene: cell_func\n", "Behind the scene of cell, there is machinary designed to make it work smoothly and transparently in most cases. However, sometimes the user may need to dig deeper. Here is an example for code that fails..." ] }, { "cell_type": "code", "execution_count": 50, "id": "environmental-samuel", "metadata": {}, "outputs": [], "source": [ "from pyg import *\n", "import pytest\n", "\n", "def twox(x):\n", " return x*2\n", "a = cell(a = 1)\n", "c = cell(twox, x = a)\n", "\n", "with pytest.raises(KeyError):\n", " c()" ] }, { "cell_type": "markdown", "id": "minute-municipality", "metadata": {}, "source": [ "c tries to run the function. The function demands parameter x. When looking at the cells provided, cell 'a' does not contain anything like 'x' so the function fails." ] }, { "cell_type": "code", "execution_count": 51, "id": "interested-ensemble", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "cell\n", "x:\n", " cell\n", " {'data': 1, 'function': None}\n", "function:\n", " \n", "data:\n", " 2" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "a = cell(data = 1)\n", "cell(twox, x = a)()" ] }, { "cell_type": "markdown", "id": "brave-serbia", "metadata": {}, "source": [ "'data' key has a preferred status so although 'x' is not in the cell, we assume but default that 'data' parameter is the one the cell wants to present to the world. This is controlled by cell_output function:" ] }, { "cell_type": "code", "execution_count": 52, "id": "productive-series", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['data']" ] }, "execution_count": 52, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cell_output(a)" ] }, { "cell_type": "code", "execution_count": 53, "id": "confused-alloy", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(['myoutput'], 3)" ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" } ], "source": [ "a = cell(data = 1, myoutput = 3, output = 'myoutput') ## you can decide your output is different\n", "cell_output(a), cell_item(a)" ] }, { "cell_type": "code", "execution_count": 54, "id": "incorrect-engagement", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "cell\n", "x:\n", " cell\n", " {'data': 1, 'myoutput': 3, 'function': None, 'output': 'myoutput'}\n", "function:\n", " \n", "data:\n", " 6" ] }, "execution_count": 54, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cell(twox, x = a)()" ] }, { "cell_type": "markdown", "id": "continuing-reunion", "metadata": {}, "source": [ "That is good but what happens if the cell has MORE than one output or we want to direct the function to grab another key?" ] }, { "cell_type": "code", "execution_count": 55, "id": "hearing-integrity", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "cell\n", "x:\n", " cell\n", " {'a': 1, 'function': None}\n", "function:\n", " cell_func\n", " relabels:\n", " {'x': 'a'}\n", " unitemized:\n", " []\n", " uncalled:\n", " []\n", " function:\n", " \n", "data:\n", " 2" ] }, "execution_count": 55, "metadata": {}, "output_type": "execute_result" } ], "source": [ "a = cell(a = 1) ## this has failed...\n", "cell(cell_func(twox, x = 'a'), x = a)() ## when you grab x, use 'a' as key" ] }, { "cell_type": "markdown", "id": "grave-emphasis", "metadata": {}, "source": [ "What if you need the cell itself rather than the items in it?" ] }, { "cell_type": "code", "execution_count": 56, "id": "weekly-ecology", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "cell\n", "x:\n", " cell\n", " {'a': 1, 'b': 2, 'function': None}\n", "function:\n", " cell_func\n", " relabels:\n", " {}\n", " unitemized:\n", " ['x']\n", " uncalled:\n", " []\n", " function:\n", " \n", "data:\n", " 3" ] }, "execution_count": 56, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def add_a_and_b(x):\n", " return x.a + x.b\n", "\n", "x = cell(a = 1, b = 2)\n", "\n", "cell(cell_func(add_a_and_b, unitemized = 'x'), x = x)()" ] }, { "cell_type": "markdown", "id": "stone-complex", "metadata": {}, "source": [ "We can see that the cell x itself is presented to the function and x.a + x.b is calculated and data == 3" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 5 }