Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

init version of online serving and rolling #290

Merged
merged 77 commits into from
May 17, 2021
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
04b916c
safe yaml loader
you-n-g Feb 16, 2021
83237ba
yml afe load
you-n-g Feb 17, 2021
1e5cf1c
init version of online serving and rolling
you-n-g Feb 26, 2021
24024d5
qlib auto init basedon project & black format
you-n-g Feb 27, 2021
c4733f6
Merge pull request #1 from you-n-g/online_srv
lzh222333 Mar 2, 2021
b84156f
Consider more situations about task_config.
lzh222333 Mar 3, 2021
05cf0e1
add task_generator method and update some hint
lzh222333 Mar 3, 2021
fd2c1ba
Update some hint
lzh222333 Mar 3, 2021
2882929
Add an example about workflow using RollingGen.
lzh222333 Mar 3, 2021
a244f87
modified the comments
lzh222333 Mar 8, 2021
def132e
modified format and added TaskCollector
lzh222333 Mar 8, 2021
83dbdfb
finished document and example
lzh222333 Mar 9, 2021
e2f5827
update task manager
lzh222333 Mar 10, 2021
2ca2071
format code
lzh222333 Mar 10, 2021
48f0fc1
first version of online serving
lzh222333 Mar 11, 2021
0df88c0
bug fixed and update collect.py
lzh222333 Mar 11, 2021
44a7dc0
update docs and fix duplicated pred bug
you-n-g Mar 12, 2021
5de7870
Merge branch 'online_srv' of github.com:you-n-g/qlib into online_srv
you-n-g Mar 12, 2021
6d8aa21
the second version of online serving
lzh222333 Mar 12, 2021
9d84d38
format code and add example
lzh222333 Mar 12, 2021
e4e8a4a
fix task name & add cur_path
you-n-g Mar 12, 2021
8362780
fix import bug
you-n-g Mar 14, 2021
646d899
update docstring and document
lzh222333 Mar 15, 2021
0bc49da
add task management to index.rst
lzh222333 Mar 15, 2021
e3730b3
more clearly structure
lzh222333 Mar 16, 2021
5953365
finished update_online_pred demo
lzh222333 Mar 16, 2021
d33041d
format example
lzh222333 Mar 16, 2021
8abdd63
online_serving V3
lzh222333 Mar 18, 2021
84d5318
Merge branch 'online_srv_wd' into online_srv
you-n-g Mar 19, 2021
d66d4ec
Merge branch 'main' into online_srv
lzh222333 Mar 23, 2021
46cd576
Online Serving V4
lzh222333 Mar 26, 2021
9bf819e
Merge branch 'online_srv' of https://github.com/you-n-g/qlib into onl…
lzh222333 Mar 26, 2021
ee45a78
Merge branch 'main' into online_srv
lzh222333 Mar 26, 2021
1f2d2c9
online debug
lzh222333 Mar 30, 2021
eae94d1
Merge remote-tracking branch 'microsoft/qlib/main' into online_srv
lzh222333 Mar 30, 2021
544365f
ensemble & get_exp & dataset_pickle
lzh222333 Mar 31, 2021
3724273
Merge remote-tracking branch 'microsoft/qlib/main' into online_srv
lzh222333 Mar 31, 2021
edcd7b1
bug fixed & code format
lzh222333 Mar 31, 2021
bd7a1c1
trainer & group & collect & ensemble
lzh222333 Apr 2, 2021
431a9c9
online serving v5
lzh222333 Apr 2, 2021
cb42e99
bug fixed & examples fire
lzh222333 Apr 7, 2021
1dbb561
Fix some API(for lb nn)
you-n-g Apr 7, 2021
7160579
Merge branch 'online_srv_wd' into online_srv
you-n-g Apr 7, 2021
c20eb5c
format code
lzh222333 Apr 8, 2021
18bf4b5
parameter adjustment
you-n-g Apr 8, 2021
a366c11
Update features for hyb nn
you-n-g Apr 9, 2021
cca43cf
Refactor update & modification when running NN
you-n-g Apr 11, 2021
b15e5e3
Fix the multi-processing bug
you-n-g Apr 12, 2021
5095b2a
simulator & examples
lzh222333 Apr 13, 2021
cec318f
online serving V7
lzh222333 Apr 16, 2021
de0a0c0
bug fixed
lzh222333 Apr 22, 2021
319396c
online serving V8
lzh222333 Apr 25, 2021
0058f7d
Online Serving V8
lzh222333 Apr 26, 2021
42f5100
update collector
lzh222333 Apr 27, 2021
36ab078
filter
Apr 28, 2021
45c6dfc
filter
Apr 28, 2021
fa4511c
filter
Apr 28, 2021
40cf83e
online serving V9 middle status
lzh222333 Apr 28, 2021
6f66934
Merge branch 'online_srv' of https://github.com/you-n-g/qlib into onl…
lzh222333 Apr 28, 2021
67c5740
OnlineServing V9
lzh222333 Apr 29, 2021
2b7ffa1
Merge remote-tracking branch 'microsoft/main' into online_srv
lzh222333 Apr 29, 2021
1c99fb3
Merge remote-tracking branch 'microsoft/main' into online_srv
lzh222333 May 6, 2021
84c56f1
docs and bug fixed
lzh222333 May 6, 2021
846c64f
fix param
binlins May 6, 2021
9dfd001
online serving v10
lzh222333 May 7, 2021
bec65dd
add document and reindex
binlins May 7, 2021
08edb92
add flt_data doc
binlins May 7, 2021
060a32e
Merge branch 'online_srv' into online_srv_blin
you-n-g May 7, 2021
1c605e5
Merge pull request #14 from you-n-g/online_srv_blin
you-n-g May 7, 2021
4c23261
Merge branch 'online_srv' of https://github.com/you-n-g/qlib into onl…
lzh222333 May 9, 2021
f5ded06
Merge remote-tracking branch 'microsoft/main' into online_srv
lzh222333 May 9, 2021
370b6aa
logger & doc
lzh222333 May 9, 2021
d71a666
Online serving V11
lzh222333 May 13, 2021
ebd01e0
Online Serving V11
lzh222333 May 14, 2021
aef3f18
format code
lzh222333 May 14, 2021
a986379
bug fixed
lzh222333 May 14, 2021
8c3a08b
Finally!
lzh222333 May 17, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions qlib/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ def init_from_yaml_conf(conf_path, **kwargs):
init(default_conf, **config)


def get_project_path(config_name="config.yaml") -> Path:
def get_project_path(config_name="config.yaml", cur_path=None) -> Path:
"""
If users are building a project follow the following pattern.
- Qlib is a sub folder in project path
Expand All @@ -181,7 +181,8 @@ def get_project_path(config_name="config.yaml") -> Path:
FileNotFoundError:
If project path is not found
"""
cur_path = Path(__file__).absolute().resolve()
if cur_path is None:
cur_path = Path(__file__).absolute().resolve()
while True:
if (cur_path / config_name).exists():
return cur_path
Expand All @@ -199,7 +200,7 @@ def auto_init(**kwargs):
"""

try:
pp = get_project_path()
pp = get_project_path(cur_path=kwargs.pop("cur_path", None))
except FileNotFoundError:
init(**kwargs)
else:
Expand Down
7 changes: 7 additions & 0 deletions qlib/data/dataset/handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,9 @@ class DataHandler(Serializable):
SH600004 13.313329 11800983.0 13.313329 13.317701 0.183632 0.0042
SH600005 37.796539 12231662.0 38.258602 37.919757 0.970325 0.0289


Tips for improving the performance of datahandler
- Fetching data with `col_set=CS_RAW` will return the raw data and may avoid pandas from copying the data when calling `loc`
"""

def __init__(
Expand Down Expand Up @@ -257,6 +260,10 @@ def get_range_iterator(
class DataHandlerLP(DataHandler):
"""
DataHandler with **(L)earnable (P)rocessor**

Tips to improving the performance of data handler
- To reduce the memory cost
- `drop_raw=True`: this will modify the data inplace on raw data;
"""

# data key
Expand Down
4 changes: 2 additions & 2 deletions qlib/workflow/task/update.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,8 +97,8 @@ def update_all_pred(self, rec_filter_func=None):

.. code-block:: python

def rec_filter_func(recorder):
task_config = recorder.load_object("task")
def record_filter(record):
task_config = record.load_object("task")
if task_config["model"]["class"]=="LGBModel":
return True
return False
Expand Down
1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
"joblib>=0.17.0",
"ruamel.yaml>=0.16.12",
"pymongo==3.7.2", # For task management
"scikit-learn>=0.22",
]

# Numpy include
Expand Down