pyg.base.cell

cell is a dict that forms part of a calculation graph. Most usefully, db_cell is implemented to maintain persistency of the function output in MongoDB. Before we start, we will show a few examples of how a cell works. Then, we will build a toy example of trading stocks based on an exponentially weighted crossover.

  • We will start by creating the system using pyg.base.dictable and pyg.timeseries.

  • We then repeat the same code, this time modifying it slightly to save the data and calculation graph in MongoDB while running the calculation.

  • We conclude by discussing the two approaches

Cell 101

[1]:

from pyg import *
a = cell(lambda x, y: x + y,  x = 1, y = 2)
b = cell(lambda x, y: x * y,  x = 2, y = a)
b

[1]:

cell
x:
    2
y:
    cell
    {'x': 1, 'y': 2, 'function': <function <lambda> at 0x000002A68A888940>}
function:
    <function <lambda> at 0x000002A68A888700>

[2]:

b.keys() ## b is a dict

[2]:

['x', 'y', 'function']

[3]:

b._args ## inputs

[3]:

['x', 'y']

[4]:

b._output ## where the output will go once we calculate it

[4]:

['data']

[5]:

assert b.run() ## b has not calculated yet... please run it

[6]:

b() # calculated object note b().data

[6]:

cell
x:
    2
y:
    cell
    x:
        1
    y:
        2
    function:
        <function <lambda> at 0x000002A68A888940>
    data:
        3
function:
    <function <lambda> at 0x000002A68A888700>
data:
    6

[7]:

assert not b().run() ## b has calculated now... no need to run it

[8]:

cell(lambda x, y: x ** y)(x = a, y = 2) # you can define the cell and then call it with the values

[8]:

cell
function:
    <function <lambda> at 0x000002A68E0EF0D0>
x:
    cell
    x:
        1
    y:
        2
    function:
        <function <lambda> at 0x000002A68A888940>
    data:
        3
y:
    2
data:
    9

Workflow without saving to the database

[9]:

from pyg import *;
import yfinance as yf # see https://github.com/ranaroussi/yfinance
constituents = dictable(read_csv('d:/dropbox/yoav/python/pyg/docs/constituents_csv.csv')).rename(lower) # downloaded from <https://datahub.io/core/s-and-p-500-companies#resource-constituents>
constituents

[9]:

dictable[505 x 3]
symbol|name                  |sector
MMM   |3M Company            |Industrials
AOS   |A.O. Smith Corp       |Industrials
ABT   |Abbott Laboratories   |Health Care
...505 rows...
ZBH   |Zimmer Biomet Holdings|Health Care
ZION  |Zions Bancorp         |Financials
ZTS   |Zoetis                |Health Care

[10]:

stocks = constituents.inc(sector = 'Energy')
stocks

[10]:

dictable[26 x 3]
name              |sector|symbol
Apache Corporation|Energy|APA
Baker Hughes Co   |Energy|BKR
Cabot Oil & Gas   |Energy|COG
...26 rows...
TechnipFMC        |Energy|FTI
Valero Energy     |Energy|VLO
Williams Companies|Energy|WMB

[11]:

stocks = stocks(history = lambda symbol, sector, name: yf.download(tickers = symbol))

[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed

1 Failed download:
- NBL: No data found, symbol may be delisted
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed

[12]:

stocks = stocks.inc(lambda history: len(history)>0)

[13]:

stocks = stocks(adj = lambda history: getitem(value = history, key = 'Adj Close'))

[14]:

stocks = stocks(rtn = lambda adj: diff(a = adj))

[15]:

stocks = stocks(vol = lambda rtn: ewmstd(a = rtn, n = 30))

[33]:

_data = 'data'
def crossover_(a, fast, slow, vol, instate = None):
    state = Dict(fast = {}, slow = {}, vol = {}) if instate is None else instate
    fast_ewma_ = ewma_(a, fast, instate = state.fast)
    slow_ewma_ = ewma_(a, slow, instate = state.slow)
    raw_signal = fast_ewma_.data - slow_ewma_.data
    signal_rms = ewmrms_(raw_signal, vol, instate = state.vol)
    normalized = raw_signal/v2na(signal_rms.data)
    return Dict(data = normalized, state = Dict(fast = fast_ewma_.state, slow = slow_ewma_.state, vol = signal_rms.state))

crossover_.output = ['data', 'state']

def crossover(a, fast, slow, vol, state = None):
    return crossover_(a, fast, slow, vol, instate = state)

some more functions to calculate the profits & loss as well as the signal/noise ratio

[17]:

def signal_pnl(signal, rtn, vol):
    return shift(signal) * (rtn/vol)

def information_ratio(pnl):
    return 16 * ts_mean(pnl) / ts_std(pnl)

[18]:

forecasts = stocks * dictable(fast = [2,4,8], slow = [6,12,24], forecast = ['fast', 'medium', 'slow'])

[19]:

forecasts = forecasts(signal = lambda rtn, fast, slow: crossover_(rtn, fast = fast, slow = slow, vol = 30).data)

[20]:

forecasts = forecasts(pnl = lambda signal, rtn, vol: signal_pnl(signal = signal, rtn = rtn, vol = vol))

[21]:

forecasts = forecasts(ir = lambda pnl: information_ratio(pnl = pnl))

[22]:

print(forecasts.pivot('symbol', 'forecast', 'ir', [last, f12]))

symbol|fast |medium|slow
APA   |0.13 |-0.03 |-0.10
BKR   |0.06 |-0.10 |-0.14
COG   |0.20 |0.12  |-0.02
COP   |-0.14|-0.17 |-0.18
CVX   |0.47 |0.30  |0.11
CXO   |0.01 |-0.14 |-0.34
DVN   |0.15 |0.15  |0.17
EOG   |0.16 |0.05  |-0.03
FANG  |0.00 |-0.06 |-0.18
FTI   |-0.18|-0.37 |-0.37
HAL   |0.73 |0.47  |0.29
HES   |0.16 |0.03  |0.00
HFC   |0.86 |0.80  |0.71
KMI   |0.45 |0.13  |0.03
MPC   |0.21 |0.45  |0.72
MRO   |0.24 |0.16  |0.14
NOV   |0.10 |-0.04 |-0.04
OKE   |-0.22|-0.15 |-0.06
OXY   |-0.23|-0.29 |-0.22
PSX   |-0.02|0.11  |0.42
PXD   |0.22 |0.17  |0.23
SLB   |-0.08|-0.25 |-0.33
VLO   |0.30 |0.29  |0.41
WMB   |0.17 |-0.08 |-0.22
XOM   |-0.03|-0.28 |-0.43

Workflow while saving to MongoDB

Table creation

We create three tables dependending on the primary keys we will be using.

[23]:

idb = partial(mongo_table, db = 'demo', table = 'items', pk = 'item')
sdb = partial(mongo_table, db = 'demo', table = 'stock', pk = ['item', 'symbol'])
fdb = partial(mongo_table, db = 'demo', table = 'forecast', pk = ['item', 'symbol', 'forecast'])

[24]:

idb().insert_one(Dict(item = 'constituents', data = constituents))

[24]:

ObjectId('602a566073531499b3861809')

Any code differences?

Most of the code remains the same as above, except:

  • We wrap it inside a periodic_cell so it is calculated daily

  • We add reference to where we want to store it in MongoDB by specifying the db as well as the primary keys of that table

  • To run the function, we need to call the cell. This: loads the cell from the database (if found), checking if it even needs running and if so, runs it.

[25]:

stocks = stocks(history = lambda symbol, sector, name: periodic_cell(yf.download, tickers = symbol,      # these are the inputs for the function
                                            db = sdb, item = 'history', symbol = symbol)())              # these define where the data goes to

2021-03-06 19:59:31,197 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'APA'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:34,627 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'BKR'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:35,356 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'COG'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:36,122 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'CVX'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:38,242 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'CXO'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:38,754 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'COP'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:39,444 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'DVN'))


[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:40,591 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'FANG'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:40,978 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'EOG'))


[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:41,557 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'XOM'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:44,056 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'HAL'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:45,237 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'HES'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:46,323 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'HFC'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:47,026 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'KMI'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:48,145 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'MRO'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:50,428 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'MPC'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:51,781 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'NOV'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:52,541 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'OXY'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:54,166 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'OKE'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:55,333 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'PSX'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:55,735 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'PXD'))


[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:56,563 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'SLB'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:58,319 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'FTI'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:58,844 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'VLO'))

[*********************100%***********************]  1 of 1 completed

2021-03-06 19:59:59,789 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('history', 'WMB'))

[*********************100%***********************]  1 of 1 completed

Accessing the data in MongoDB

The data is now in the database and can be accessed:

[26]:

get_data('stock', 'demo', symbol = 'MMM')

[26]:
Open High Low Close Adj Close Volume
Date
1970-01-02 6.851563 6.890625 6.843750 6.851563 1.471232 72000
1970-01-05 6.859375 6.898438 6.859375 6.890625 1.479620 446400
1970-01-06 6.890625 6.960938 6.882813 6.960938 1.494717 176000
1970-01-07 6.960938 7.015625 6.945313 7.000000 1.503106 164800
1970-01-08 7.000000 7.109375 6.984375 7.093750 1.523237 304000
... ... ... ... ... ... ...
2021-02-08 179.300003 180.869995 179.169998 180.759995 179.282608 2355100
2021-02-09 181.220001 181.899994 180.179993 180.940002 179.461151 1942800
2021-02-10 181.880005 182.380005 180.639999 181.080002 179.600006 1929000
2021-02-11 179.350006 179.880005 175.839996 177.210007 177.210007 2187100
2021-02-12 177.270004 178.839996 177.210007 178.699997 178.699997 1081500

12895 rows × 6 columns

[27]:

stocks = stocks.inc(lambda history: len(history.data)>0)

[28]:

stocks = stocks(adj = lambda history, symbol: periodic_cell(getitem, value = history, key = 'Adj Close',
                                                            db = sdb, symbol = symbol, item = 'adj')())

2021-03-06 20:00:02,042 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'APA'))
2021-03-06 20:00:03,128 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'BKR'))
2021-03-06 20:00:03,468 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'COG'))
2021-03-06 20:00:03,773 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'CVX'))
2021-03-06 20:00:04,094 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'CXO'))
2021-03-06 20:00:04,340 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'COP'))
2021-03-06 20:00:04,633 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'DVN'))
2021-03-06 20:00:04,936 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'FANG'))
2021-03-06 20:00:05,160 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'EOG'))
2021-03-06 20:00:05,518 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'XOM'))
2021-03-06 20:00:05,860 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'HAL'))
2021-03-06 20:00:06,272 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'HES'))
2021-03-06 20:00:06,881 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'HFC'))
2021-03-06 20:00:07,289 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'KMI'))
2021-03-06 20:00:07,601 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'MRO'))
2021-03-06 20:00:07,905 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'MPC'))
2021-03-06 20:00:08,608 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'NOV'))
2021-03-06 20:00:09,233 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'OXY'))
2021-03-06 20:00:09,762 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'OKE'))
2021-03-06 20:00:10,238 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'PSX'))
2021-03-06 20:00:11,431 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'PXD'))
2021-03-06 20:00:13,090 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'SLB'))
2021-03-06 20:00:14,979 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'FTI'))
2021-03-06 20:00:15,767 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'VLO'))
2021-03-06 20:00:16,145 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('adj', 'WMB'))

[29]:

stocks = stocks(rtn = lambda adj, symbol: periodic_cell(diff, a = adj,
                                                        db = sdb, symbol = symbol, item = 'rtn')())

2021-03-06 20:00:16,480 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'APA'))
2021-03-06 20:00:16,930 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'BKR'))
2021-03-06 20:00:17,348 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'COG'))
2021-03-06 20:00:17,656 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'CVX'))
2021-03-06 20:00:18,515 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'CXO'))
2021-03-06 20:00:18,816 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'COP'))
2021-03-06 20:00:19,750 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'DVN'))
2021-03-06 20:00:20,084 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'FANG'))
2021-03-06 20:00:20,322 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'EOG'))
2021-03-06 20:00:20,813 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'XOM'))
2021-03-06 20:00:21,168 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'HAL'))
2021-03-06 20:00:21,636 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'HES'))
2021-03-06 20:00:22,050 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'HFC'))
2021-03-06 20:00:22,342 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'KMI'))
2021-03-06 20:00:22,764 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'MRO'))
2021-03-06 20:00:23,127 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'MPC'))
2021-03-06 20:00:23,383 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'NOV'))
2021-03-06 20:00:23,681 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'OXY'))
2021-03-06 20:00:23,978 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'OKE'))
2021-03-06 20:00:24,327 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'PSX'))
2021-03-06 20:00:24,580 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'PXD'))
2021-03-06 20:00:24,850 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'SLB'))
2021-03-06 20:00:25,313 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'FTI'))
2021-03-06 20:00:25,611 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'VLO'))
2021-03-06 20:00:25,938 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('rtn', 'WMB'))

[30]:

stocks = stocks(vol = lambda rtn, symbol: periodic_cell(ewmstd, a = rtn, n =  30,
                                                        db = sdb, symbol = symbol, item = 'vol')())

2021-03-06 20:00:26,300 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'APA'))
2021-03-06 20:00:26,595 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'BKR'))
2021-03-06 20:00:26,844 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'COG'))
2021-03-06 20:00:27,498 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'CVX'))
2021-03-06 20:00:28,895 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'CXO'))
2021-03-06 20:00:29,513 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'COP'))
2021-03-06 20:00:30,345 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'DVN'))
2021-03-06 20:00:31,002 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'FANG'))
2021-03-06 20:00:31,378 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'EOG'))
2021-03-06 20:00:31,711 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'XOM'))
2021-03-06 20:00:34,153 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'HAL'))
2021-03-06 20:00:35,635 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'HES'))
2021-03-06 20:00:36,165 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'HFC'))
2021-03-06 20:00:36,492 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'KMI'))
2021-03-06 20:00:37,163 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'MRO'))
2021-03-06 20:00:37,823 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'MPC'))
2021-03-06 20:00:38,481 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'NOV'))
2021-03-06 20:00:39,119 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'OXY'))
2021-03-06 20:00:39,425 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'OKE'))
2021-03-06 20:00:39,705 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'PSX'))
2021-03-06 20:00:40,127 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'PXD'))
2021-03-06 20:00:40,378 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'SLB'))
2021-03-06 20:00:40,631 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'FTI'))
2021-03-06 20:00:41,056 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'VLO'))
2021-03-06 20:00:41,304 - pyg - INFO - ('localhost', 27017, 'demo', 'stock', ('item', 'symbol'), ('vol', 'WMB'))

Calculating the forecasts & saving them

[31]:

forecasts = stocks * dictable(fast = [2,4,8], slow = [6,12,24], forecast = ['fast', 'medium', 'slow'])

[34]:

forecasts = forecasts(signal = lambda rtn, fast, slow, symbol, forecast: periodic_cell(crossover_, a = rtn, fast = fast, slow = slow, vol = 30,
                                            db = fdb, symbol = symbol, forecast = forecast, item = 'signal')())

2021-03-06 20:04:15,290 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'APA'))
2021-03-06 20:04:52,869 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'APA'))
2021-03-06 20:04:53,186 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'APA'))
2021-03-06 20:04:53,434 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'BKR'))
2021-03-06 20:04:53,681 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'BKR'))
2021-03-06 20:04:53,913 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'BKR'))
2021-03-06 20:04:54,203 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'COG'))
2021-03-06 20:04:56,089 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'COG'))
2021-03-06 20:04:56,727 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'COG'))
2021-03-06 20:04:57,250 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'CVX'))
2021-03-06 20:04:58,763 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'CVX'))
2021-03-06 20:04:59,116 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'CVX'))
2021-03-06 20:04:59,518 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'CXO'))
2021-03-06 20:05:00,010 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'CXO'))
2021-03-06 20:05:00,371 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'CXO'))
2021-03-06 20:05:00,771 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'COP'))
2021-03-06 20:05:01,088 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'COP'))
2021-03-06 20:05:01,374 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'COP'))
2021-03-06 20:05:01,673 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'DVN'))
2021-03-06 20:05:01,948 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'DVN'))
2021-03-06 20:05:02,256 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'DVN'))
2021-03-06 20:05:02,500 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'FANG'))
2021-03-06 20:05:02,708 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'FANG'))
2021-03-06 20:05:02,927 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'FANG'))
2021-03-06 20:05:03,134 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'EOG'))
2021-03-06 20:05:03,417 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'EOG'))
2021-03-06 20:05:03,671 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'EOG'))
2021-03-06 20:05:03,898 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'XOM'))
2021-03-06 20:05:04,224 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'XOM'))
2021-03-06 20:05:04,485 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'XOM'))
2021-03-06 20:05:04,701 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'HAL'))
2021-03-06 20:05:05,394 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'HAL'))
2021-03-06 20:05:05,652 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'HAL'))
2021-03-06 20:05:05,902 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'HES'))
2021-03-06 20:05:06,130 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'HES'))
2021-03-06 20:05:06,390 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'HES'))
2021-03-06 20:05:06,616 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'HFC'))
2021-03-06 20:05:06,883 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'HFC'))
2021-03-06 20:05:07,099 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'HFC'))
2021-03-06 20:05:07,321 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'KMI'))
2021-03-06 20:05:07,512 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'KMI'))
2021-03-06 20:05:07,718 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'KMI'))
2021-03-06 20:05:07,941 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'MRO'))
2021-03-06 20:05:08,197 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'MRO'))
2021-03-06 20:05:08,458 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'MRO'))
2021-03-06 20:05:08,687 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'MPC'))
2021-03-06 20:05:08,905 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'MPC'))
2021-03-06 20:05:09,129 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'MPC'))
2021-03-06 20:05:09,348 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'NOV'))
2021-03-06 20:05:09,620 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'NOV'))
2021-03-06 20:05:09,845 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'NOV'))
2021-03-06 20:05:10,052 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'OXY'))
2021-03-06 20:05:10,383 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'OXY'))
2021-03-06 20:05:10,705 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'OXY'))
2021-03-06 20:05:10,958 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'OKE'))
2021-03-06 20:05:11,232 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'OKE'))
2021-03-06 20:05:11,475 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'OKE'))
2021-03-06 20:05:11,715 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'PSX'))
2021-03-06 20:05:11,997 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'PSX'))
2021-03-06 20:05:12,207 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'PSX'))
2021-03-06 20:05:12,414 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'PXD'))
2021-03-06 20:05:12,669 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'PXD'))
2021-03-06 20:05:12,901 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'PXD'))
2021-03-06 20:05:13,128 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'SLB'))
2021-03-06 20:05:13,751 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'SLB'))
2021-03-06 20:05:13,985 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'SLB'))
2021-03-06 20:05:14,230 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'FTI'))
2021-03-06 20:05:14,503 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'FTI'))
2021-03-06 20:05:14,775 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'FTI'))
2021-03-06 20:05:15,037 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'VLO'))
2021-03-06 20:05:15,346 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'VLO'))
2021-03-06 20:05:15,578 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'VLO'))
2021-03-06 20:05:15,814 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'signal', 'WMB'))
2021-03-06 20:05:16,067 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'signal', 'WMB'))
2021-03-06 20:05:16,283 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'signal', 'WMB'))

[36]:

forecasts = forecasts(pnl = lambda signal, rtn, vol, symbol, forecast: periodic_cell(signal_pnl, signal = signal, rtn = rtn, vol = vol,
                                                        db = fdb, symbol = symbol, forecast = forecast, item = 'pnl')())

2021-03-06 20:07:41,567 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'APA'))
2021-03-06 20:09:01,609 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'APA'))
2021-03-06 20:09:02,368 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'APA'))
2021-03-06 20:09:03,885 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'BKR'))
2021-03-06 20:09:04,804 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'BKR'))
2021-03-06 20:09:05,484 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'BKR'))
2021-03-06 20:09:06,030 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'COG'))
2021-03-06 20:09:06,627 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'COG'))
2021-03-06 20:09:07,453 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'COG'))
2021-03-06 20:09:08,153 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'CVX'))
2021-03-06 20:09:09,125 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'CVX'))
2021-03-06 20:09:09,730 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'CVX'))
2021-03-06 20:09:10,582 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'CXO'))
2021-03-06 20:09:11,216 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'CXO'))
2021-03-06 20:09:11,904 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'CXO'))
2021-03-06 20:09:12,350 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'COP'))
2021-03-06 20:09:13,415 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'COP'))
2021-03-06 20:09:14,255 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'COP'))
2021-03-06 20:09:15,747 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'DVN'))
2021-03-06 20:09:16,408 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'DVN'))
2021-03-06 20:09:17,055 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'DVN'))
2021-03-06 20:09:17,802 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'FANG'))
2021-03-06 20:09:18,207 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'FANG'))
2021-03-06 20:09:18,601 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'FANG'))
2021-03-06 20:09:20,074 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'EOG'))
2021-03-06 20:09:20,820 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'EOG'))
2021-03-06 20:09:21,605 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'EOG'))
2021-03-06 20:09:23,377 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'XOM'))
2021-03-06 20:09:25,407 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'XOM'))
2021-03-06 20:09:27,902 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'XOM'))
2021-03-06 20:09:29,626 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'HAL'))
2021-03-06 20:09:31,617 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'HAL'))
2021-03-06 20:09:32,908 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'HAL'))
2021-03-06 20:09:34,001 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'HES'))
2021-03-06 20:09:35,216 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'HES'))
2021-03-06 20:09:35,767 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'HES'))
2021-03-06 20:09:36,840 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'HFC'))
2021-03-06 20:09:38,454 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'HFC'))
2021-03-06 20:09:38,896 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'HFC'))
2021-03-06 20:09:39,445 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'KMI'))
2021-03-06 20:09:40,384 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'KMI'))
2021-03-06 20:09:40,983 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'KMI'))
2021-03-06 20:09:41,528 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'MRO'))
2021-03-06 20:09:43,164 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'MRO'))
2021-03-06 20:09:44,007 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'MRO'))
2021-03-06 20:09:44,933 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'MPC'))
2021-03-06 20:09:45,778 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'MPC'))
2021-03-06 20:09:46,376 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'MPC'))
2021-03-06 20:09:46,926 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'NOV'))
2021-03-06 20:09:47,990 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'NOV'))
2021-03-06 20:09:48,697 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'NOV'))
2021-03-06 20:09:49,380 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'OXY'))
2021-03-06 20:09:50,123 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'OXY'))
2021-03-06 20:09:51,039 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'OXY'))
2021-03-06 20:09:52,698 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'OKE'))
2021-03-06 20:09:54,721 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'OKE'))
2021-03-06 20:09:55,441 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'OKE'))
2021-03-06 20:09:56,279 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'PSX'))
2021-03-06 20:09:56,974 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'PSX'))
2021-03-06 20:09:57,510 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'PSX'))
2021-03-06 20:09:57,980 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'PXD'))
2021-03-06 20:09:58,568 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'PXD'))
2021-03-06 20:09:59,271 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'PXD'))
2021-03-06 20:10:00,220 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'SLB'))
2021-03-06 20:10:02,318 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'SLB'))
2021-03-06 20:10:03,380 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'SLB'))
2021-03-06 20:10:04,161 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'FTI'))
2021-03-06 20:10:04,744 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'FTI'))
2021-03-06 20:10:05,535 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'FTI'))
2021-03-06 20:10:06,195 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'VLO'))
2021-03-06 20:10:07,851 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'VLO'))
2021-03-06 20:10:09,025 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'VLO'))
2021-03-06 20:10:10,054 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('fast', 'pnl', 'WMB'))
2021-03-06 20:10:11,128 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('medium', 'pnl', 'WMB'))
2021-03-06 20:10:11,790 - pyg - INFO - ('localhost', 27017, 'demo', 'forecast', ('forecast', 'item', 'symbol'), ('slow', 'pnl', 'WMB'))

[37]:

forecasts = forecasts(ir = lambda pnl: information_ratio(pnl = pnl.data))

[38]:

print(forecasts.pivot('symbol', 'forecast', 'ir', [last, f12]))

symbol|fast |medium|slow
APA   |0.13 |-0.03 |-0.10
BKR   |0.06 |-0.10 |-0.14
COG   |0.20 |0.12  |-0.02
COP   |-0.14|-0.17 |-0.18
CVX   |0.47 |0.30  |0.11
CXO   |0.01 |-0.14 |-0.34
DVN   |0.15 |0.15  |0.17
EOG   |0.16 |0.05  |-0.03
FANG  |0.00 |-0.06 |-0.18
FTI   |-0.18|-0.37 |-0.37
HAL   |0.73 |0.47  |0.29
HES   |0.16 |0.03  |0.00
HFC   |0.86 |0.80  |0.71
KMI   |0.45 |0.13  |0.03
MPC   |0.21 |0.45  |0.72
MRO   |0.24 |0.16  |0.14
NOV   |0.10 |-0.04 |-0.04
OKE   |-0.22|-0.15 |-0.06
OXY   |-0.23|-0.29 |-0.22
PSX   |-0.02|0.11  |0.42
PXD   |0.22 |0.17  |0.23
SLB   |-0.08|-0.25 |-0.33
VLO   |0.30 |0.29  |0.41
WMB   |0.17 |-0.08 |-0.22
XOM   |-0.03|-0.28 |-0.43

Accessing & running the graph once the graph has been created

We can access the data or the cell:

[39]:

get_cell('forecast', 'demo', symbol = 'APA', forecast = 'fast', item = 'signal')

[39]:

periodic_cell
updated:
    2021-03-06 20:04:51.410000
_period:
    1b
db:
    functools.partial(<function mongo_table at 0x000002A68D03BCA0>, db='demo', table='forecast', pk=['item', 'symbol', 'forecast'])
_id:
    602a86fb4de6ffaf1c045c8f
_pk:
    ['forecast', 'item', 'symbol']
a:
    periodic_cell
    updated:
        None
    _period:
        1b
    db:
        functools.partial(<function mongo_table at 0x000002A68D03BCA0>, db='demo', table='stock', pk=['item', 'symbol'])
    item:
        rtn
    symbol:
        APA
    function:
        None
data:
    Date
    1979-05-15         NaN
    1979-05-16         NaN
    1979-05-17   -1.402762
    1979-05-18   -0.939334
    1979-05-21   -1.194304
                    ...
    2021-03-01   -1.018576
    2021-03-02   -0.831244
    2021-03-03   -0.187666
    2021-03-04    0.972348
    2021-03-05    2.674065
    Length: 10543, dtype: float64
fast:
    2
forecast:
    fast
instate:
    None
item:
    signal
slow:
    6
state:
    Dict
    fast:
        Dict
        state:
            {'t': nan, 't0': 0.9999999999999999, 't1': 0.46353936200681184}
        t:
            nan
        t0:
            0.9999999999999999
        t1:
            1.0413438705045697
    slow:
        Dict
        state:
            {'t': nan, 't0': 0.9999999999999997, 't1': 0.2672723621042594}
        t:
            nan
        t0:
            0.9999999999999997
        t1:
            0.5552029895351339
    vol:
        Dict
        state:
            {'t': nan, 't0': 0.9999999999999983, 't2': 0.02845240938173902}
        t:
            nan
        t0:
            0.9999999999999983
        t2:
            0.033050680992027265
symbol:
    APA
vol:
    30
function:
    <function crossover_ at 0x000002A68E366C10>

And now that the graph has been created, you can actually trigger it just by loading. i.e. The code below will give you the fast signal for APA and will ensure it is up-to-date too:

[40]:

c = get_cell('forecast', 'demo', symbol = 'APA', forecast = 'fast', item = 'signal')
c = c.go()
print(c.data)

Date
1979-05-15         NaN
1979-05-16         NaN
1979-05-17   -1.402762
1979-05-18   -0.939334
1979-05-21   -1.194304
                ...
2021-03-01   -1.018576
2021-03-02   -0.831244
2021-03-03   -0.187666
2021-03-04    0.972348
2021-03-05    2.674065
Length: 10543, dtype: float64

Point-in-time, cache and persistency

The pk-tables save a full history of all your data. To avoid hitting the database all the time, we also have a local GRAPH singleton that caches all the cells by their address. The cell has few basic operations we need to understand:

  • cell.run(): Returns True/False if the cell needs to be calculated. db_cell() just check for values in its output, periodic_cell will also check if it is a new business day.

  • cell.go(): This calculates the cell and saves the result to the database (and to GRAPH). The cell itself is not loaded but all its inputs are loaded

    • cell.go(0) : calculate only if there is a need

    • cell.go(1) : calculate me but my parents only if there is a need

    • cell.go(2) : calculate me & my parents but my grandparents only if there is a need

    • cell.go(-1) : calculate everything

  • cell.load(): This loads the data from GRAPH, if not in GRAPH, loads it from MongoDB (and updates also the GRAPH)

    • cell.load(-1) : Clear the data from the GRAPH

    • cell.load(0) : Load & update me from GRAPH, if not, from MongoDB, if not, just return good old me

    • cell.load(1) : If you cannot find me, throw an Exception

    • cell.load(date) : Load my version as valid on date. If none exists, throw.

    • cell.load([value]): Force GRAPH to clear, only load from DB. Same as cell.load(-1).load(value)

  • cell(): This loads & then go

[51]:

from pyg import *; from functools import partial
db = partial(mongo_table, db = 'demo', table = 'persistency', pk = 'key')
db().raw.drop()

def f(a, b):
    return a+b

2021-03-19 08:42:10,778 - pyg - INFO - INFO: deleting 23 documents based on M{}

[56]:

## now we set up a fake calculation tree:
x = db_cell(f, a = 1, b = 2, db = db, key = 'x')
y = db_cell(f, a = x, b = 2, db = db, key = 'y')
z = db_cell(f, a = x, b = y, db = db, key = 'z')

## and run it by running the final value we want
z = z()

[57]:

## we can access the data:
get_data('persistency', 'demo', key = 'x')

[57]:

3

[58]:

t0 = dt()  ## first breakpoint

[60]:

x = db_cell(f, a = 10, b = 20, db = db, key = 'x').go()
y = db_cell(f, a = x,  b = 20, db = db, key = 'y').go()
z = db_cell(f, a = x,  b = y, db = db, key = 'z').go()

2021-03-19 08:43:07,223 - pyg - INFO - ('localhost', 27017, 'demo', 'persistency', ('key',), ('x',))
2021-03-19 08:43:07,317 - pyg - INFO - ('localhost', 27017, 'demo', 'persistency', ('key',), ('y',))
2021-03-19 08:43:07,547 - pyg - INFO - ('localhost', 27017, 'demo', 'persistency', ('key',), ('z',))

[61]:

## and here is the new data
get_data('persistency', 'demo', key = 'x')

[61]:

30

[62]:

## and here is the data valid at our first breakpoint. get_data/get_cell always go to the database to load values
get_data('persistency', 'demo', key = 'x', _deleted = t0)

[62]:

3

We can ask a cell to load itself, but remember: it will go to GRAPH first by default. The GRAPH has only one copy of the cell, while in MongoDB, every time we recalculate/save a new version of the cell, we mark the old version in the database as “deleted” but otherwise keep it. To force a cell to load itself from the database, encase the breakpoint in a list…

[63]:

db_cell(db = db, key = 'x').load([t0])

[63]:

db_cell
db:
    functools.partial(<function mongo_table at 0x000001A9D35D43A0>, db='demo', table='persistency', pk='key')
_id:
    60546402e49a0e77dca6dff3
_pk:
    ['key']
a:
    1
b:
    2
data:
    3
key:
    x
_deleted:
    2021-03-19 08:42:42.668000
function:
    <function f at 0x000001A9D5C4DDC0>

We can force a full recalculation of the tree in a single line of code:

[82]:

db_cell(db = db, key = 'z')(go = -1, mode = [t0]).data ## Should be 8, same as the old value

2021-03-19 08:58:39,317 - pyg - INFO - ('localhost', 27017, 'demo', 'persistency', ('key',), ('z',))
2021-03-19 08:58:39,946 - pyg - INFO - ('localhost', 27017, 'demo', 'persistency', ('key',), ('x',))
2021-03-19 08:58:40,087 - pyg - INFO - ('localhost', 27017, 'demo', 'persistency', ('key',), ('y',))
2021-03-19 08:58:40,213 - pyg - INFO - ('localhost', 27017, 'demo', 'persistency', ('key',), ('x',))

[82]:

8

Comparison of the two workflows

Saving to the database has negatives:

  • does require some (but really not much) additional code to specify where each data item goes to

  • slows down the calculation

Conversely,

  • We get full persistency: We can access each part of the graph with full visibility on the inputs, the function used to calculate the result, the function output(s), the location of where the data is stored and the time it was last updated as well as the periodicity it is calculated.

  • We get full audit, past calculations remain available to track (and indeed, rerun) if anything goes wrong

  • Each node will manage its schedule, ensuring data is up-to-date

  • We can run just the parts of the graph we are interested in (and can run in parallel)

To save or not to save?

Luckily we don’t really need to decide on one workflow or the other as both can happily coexist. We can build a calculation graph and decide that some key points in the calculation we want to save while intermediate calculations we can calculate on the fly and not save at all. We have met the crossover function. Here we implement it ‘on the fly’ while saving just final value to db

[1]:

from pyg import *; import pandas as pd; import numpy as np; from functools import partial

[2]:

def fake_ts(ticker):
    return pd.Series(np.random.normal(0,1,1000), drange(-999))
db = partial(mongo_table, db = 'test', table = 'test', pk = ['key'])
db().raw.drop()

2021-03-14 00:08:03,885 - pyg - INFO - INFO: deleting 7 documents based on M{}

[2]:

<class 'pyg.mongo._cursor.mongo_cursor'> for Collection(Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True), 'test'), 'test')
M{} None
documents count: 0

[3]:

appl = db_cell(fake_ts, ticker = 'appl', key = 'appl_rtn', db = db)()

#I am never saving these, In fact, I don't want to see these in calculation log.
a = cell(ewma, a = appl, n = 30)
b = cell(ewma, a = appl, n = 50)

# I may want to save these nodes but haven't made up my mind
# I do want to see the calculations in the log though...
# I replace db by the primary keys of table (here 'key').
# This allows as to see the calculation log as it happens.
# data is not saved to db though until I switch to db = db as opposed to db = 'key'

c = db_cell(sub_, a = a, b = b, key = 'calculate difference of ewma', db = 'key')
d = db_cell(ewmrms, a = c, n = 100, key = 'root mean square of difference', db = 'key')

# The final crossover I definitely want to save to db:
final_value = db_cell(div_, a = c, b = d, key = 'appl_crossover', db = db)()

2021-03-14 00:08:04,243 - pyg - INFO - ('localhost', 27017, 'test', 'test', ('key',), ('appl_rtn',))
2021-03-14 00:08:04,416 - pyg - INFO - ('localhost', 27017, 'test', 'test', ('key',), ('appl_crossover',))
2021-03-14 00:08:04,419 - pyg - INFO - (None, None, None, None, ('key',), ('calculate difference of ewma',))
2021-03-14 00:08:05,162 - pyg - INFO - (None, None, None, None, ('key',), ('root mean square of difference',))
2021-03-14 00:08:05,164 - pyg - INFO - (None, None, None, None, ('key',), ('calculate difference of ewma',))

[4]:

db().key

[4]:

['appl_crossover', 'appl_rtn']

So although we had several intermediate steps, we decided to save just the final crossover in the database, If we look at the inputs for the function, you can see that the values are not saved in the database, though the full calculation tree is. Therefore, one can reload the node and then recalculate all the intermediate values on the fly

[5]:

loaded_and_recalculated = (db()[dict(key = 'appl_crossover')] - 'data').go(1) ## but once recalculated, we can assert we got the same result
assert eq(loaded_and_recalculated.data, final_value.data)

2021-03-14 00:08:09,934 - pyg - INFO - ('localhost', 27017, 'test', 'test', ('key',), ('appl_crossover',))
2021-03-14 00:08:09,936 - pyg - INFO - (None, None, None, None, ('key',), ('calculate difference of ewma',))
2021-03-14 00:08:10,091 - pyg - INFO - (None, None, None, None, ('key',), ('root mean square of difference',))
2021-03-14 00:08:10,095 - pyg - INFO - (None, None, None, None, ('key',), ('calculate difference of ewma',))

Behind the scene: cell_func

Behind the scene of cell, there is machinary designed to make it work smoothly and transparently in most cases. However, sometimes the user may need to dig deeper. Here is an example for code that fails…

[50]:

from pyg import *
import pytest

def twox(x):
    return x*2
a = cell(a = 1)
c = cell(twox, x = a)

with pytest.raises(KeyError):
    c()

c tries to run the function. The function demands parameter x. When looking at the cells provided, cell ‘a’ does not contain anything like ‘x’ so the function fails.

[51]:

a = cell(data = 1)
cell(twox, x = a)()

[51]:

cell
x:
    cell
    {'data': 1, 'function': None}
function:
    <function twox at 0x000002A68A888430>
data:
    2

‘data’ key has a preferred status so although ‘x’ is not in the cell, we assume but default that ‘data’ parameter is the one the cell wants to present to the world. This is controlled by cell_output function:

[52]:

cell_output(a)

[52]:

['data']

[53]:

a = cell(data = 1, myoutput = 3, output = 'myoutput') ## you can decide your output is different
cell_output(a), cell_item(a)

[53]:

(['myoutput'], 3)

[54]:

cell(twox, x = a)()

[54]:

cell
x:
    cell
    {'data': 1, 'myoutput': 3, 'function': None, 'output': 'myoutput'}
function:
    <function twox at 0x000002A68A888430>
data:
    6

That is good but what happens if the cell has MORE than one output or we want to direct the function to grab another key?

[55]:

a = cell(a = 1) ## this has failed...
cell(cell_func(twox, x = 'a'), x = a)() ## when you grab x, use 'a' as key

[55]:

cell
x:
    cell
    {'a': 1, 'function': None}
function:
    cell_func
    relabels:
        {'x': 'a'}
    unitemized:
        []
    uncalled:
        []
    function:
        <function twox at 0x000002A68A888430>
data:
    2

What if you need the cell itself rather than the items in it?

[56]:

def add_a_and_b(x):
    return x.a + x.b

x = cell(a = 1, b = 2)

cell(cell_func(add_a_and_b, unitemized = 'x'), x = x)()

[56]:

cell
x:
    cell
    {'a': 1, 'b': 2, 'function': None}
function:
    cell_func
    relabels:
        {}
    unitemized:
        ['x']
    uncalled:
        []
    function:
        <function add_a_and_b at 0x000002A6912A3160>
data:
    3

We can see that the cell x itself is presented to the function and x.a + x.b is calculated and data == 3