feat(windows): beta support #3290

deepankarm · 2021-08-30T09:23:25Z

Docs deployed at https://feat-windows-beta--jina-docs.netlify.app/

codecov · 2021-08-30T09:32:09Z

Codecov Report

Merging #3290 (806dcd7) into master (39242dd) will increase coverage by 0.92%.
The diff coverage is 83.05%.

@@            Coverage Diff             @@
##           master    #3290      +/-   ##
==========================================
+ Coverage   89.35%   90.28%   +0.92%     
==========================================
  Files         151      151              
  Lines       11014    11047      +33     
==========================================
+ Hits         9842     9974     +132     
+ Misses       1172     1073      -99

Flag	Coverage Δ
daemon	`45.70% <50.84%> (-0.05%)`	⬇️
jina	`90.25% <83.05%> (+0.94%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
jina/hubble/hubio.py	`87.50% <0.00%> (-0.28%)`	⬇️
jina/__init__.py	`71.25% <66.66%> (+0.36%)`	⬆️
jina/logging/profile.py	`98.18% <66.66%> (-0.59%)`	⬇️
jina/helper.py	`83.12% <73.33%> (+1.23%)`	⬆️
jina/logging/logger.py	`92.63% <83.33%> (-0.85%)`	⬇️
jina/types/arrays/memmap.py	`95.03% <87.50%> (-0.23%)`	⬇️
jina/flow/base.py	`90.03% <100.00%> (+1.55%)`	⬆️
jina/helloworld/multimodal/app.py	`86.79% <100.00%> (+0.51%)`	⬆️
jina/importer.py	`87.17% <100.00%> (ø)`
jina/peapods/peas/__init__.py	`90.38% <100.00%> (+4.48%)`	⬆️
... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 97519df...806dcd7. Read the comment docs.

github-actions · 2021-08-30T09:33:24Z

Latency summary

Current PR yields:

🐎🐎🐎🐎 index QPS at 1194, delta to last 2 avg.: +11%
🐎🐎 query QPS at 48, delta to last 2 avg.: +8%
🐎🐎 dam extend QPS at 44887, delta to last 2 avg.: +12%
🐎🐎 avg flow time within 1.6038 seconds, delta to last 2 avg.: -4%
🐢🐢 import jina within 0.3859 seconds, delta to last 2 avg.: -12%

Breakdown

Version	Index QPS	Query QPS	DAM Extend QPS	Avg Flow Time (s)	Import Time (s)
current	1194	48	44887	1.6038	0.3859
`2.1.3`	1108	46	42990	1.6358	0.4305
`2.1.2`	1027	42	36736	1.7097	0.4542

Backed by latency-tracking. Further commits will update this comment.

deepankarm · 2021-08-30T09:46:17Z

jina/helloworld/multimodal/flow-index.yml

@@ -7,19 +7,19 @@ executors:
      metas:
        workspace: $HW_WORKDIR
        py_modules:
-          - my_executors.py
+          - $APP_DIR/my_executors.py


This is done to solve the issue with yaml register_cls / pymodule imports in a spawned process. Should be reverted

…nto feat-windows-beta

deepankarm · 2021-09-23T04:17:06Z

jina/types/arrays/memmap.py

@@ -17,6 +17,9 @@

 import numpy as np

+from ... import __windows__
+
+__windows__ = True


@alaeddine-13 this is left from debugging?

Oh sorry, will clean it

deepankarm · 2021-09-23T04:19:01Z

jina/types/arrays/memmap.py

+        if __windows__:
+            self._body.seek(self._start)


Should this be here?

Yes, actually mmap seeks _body to 0 on windows

hanxiao · 2021-09-25T07:41:15Z

jina/__init__.py

@@ -42,7 +44,7 @@ def _warning_on_one_line(message, category, filename, lineno, *args, **kwargs):

    _set_start_method(_start_method.lower())
    _warnings.warn(f'multiprocessing start method is set to `{_start_method.lower()}`')
-    _os.unsetenv('JINA_MP_START_METHOD')
+    _os.environ.pop('JINA_MP_START_METHOD')


but is this really the same as unsetenv? isn't dict.pop modify the dict object itself instead of really unset env vars?

This seems to do the same thing as unsetenv (tested on Ubuntu, python 3.7.11)

import os import subprocess echo = lambda x, y: print(x, '\t', subprocess.check_output([f'echo ${y}'], shell=True, universal_newlines=True).strip()) def _pop(): var_name = 'FOO_POP' os.environ[var_name] = 'BAR' echo('after set env: ', var_name) os.environ.pop(var_name) echo('after os.environ.pop: ', var_name) def _unsetenv(): var_name = 'FOO_UNSET' os.environ[var_name] = 'BAR' echo('after set env: ', var_name) os.unsetenv(var_name) echo('after os.unsetenv: ', var_name) def _del(): var_name = 'FOO_DEL' os.environ[var_name] = 'BAR' echo('after set env: ', var_name) del os.environ[var_name] echo('after del os.environ: ', var_name) _pop() _unsetenv() _del()

after set env: BAR after os.environ.pop: after set env: BAR after os.unsetenv: after set env: BAR after del os.environ:

interesting! thanks for explaining it

hanxiao · 2021-09-25T07:55:12Z

jina/flow/base.py

@@ -1036,7 +1036,7 @@ def __exit__(self, exc_type, exc_val, exc_tb):
        # unset all envs to avoid any side-effect
        if self.args.env:
            for k in self.args.env.keys():
-                os.unsetenv(k)
+                os.environ.pop(k, None)


same here as above

hanxiao · 2021-09-25T08:07:04Z

jina/importer.py

            Affects:
-            - Either, change your code from using `from {user_module_name} import ...` 
+            - Either, change your code from using `from {user_module_name} import ...`


out of the scope of this PR, but im not really sure this warning is still valid @tadejsv

hanxiao

some minor changes

github-actions · 2021-09-26T05:35:19Z

📝 Docs are deployed on https://feat-windows-beta--jina-docs.netlify.app 🎉

feat: windows support (alpha)

8f6a93a

deepankarm requested review from hanxiao and a team as code owners August 30, 2021 09:23

deepankarm requested a review from Roshanjossey August 30, 2021 09:23

Merge branch 'master' into feat-windows-beta

8b5e904

deepankarm marked this pull request as draft August 30, 2021 09:32

Merge branch 'master' into feat-windows-beta

94f131b

github-actions bot added the component/flow label Aug 30, 2021

deepankarm commented Aug 30, 2021

View reviewed changes

deepankarm force-pushed the feat-windows-beta branch from abcf5bd to 2f6b633 Compare August 30, 2021 09:52

Merge branch 'feat-windows-beta' of https://github.com/jina-ai/jina i…

71d940d

…nto feat-windows-beta

deepankarm force-pushed the feat-windows-beta branch from 2f6b633 to 71d940d Compare August 30, 2021 09:58

ci: add system tests to windows

7256b6c

github-actions bot added area/cicd This issue/PR affects the cicd pipeline area/housekeeping This issue/PR is housekeeping labels Aug 30, 2021

deepankarm force-pushed the feat-windows-beta branch from 7256b6c to 52e276a Compare August 30, 2021 11:27

github-actions bot added area/setup This issue/PR affects setting up Jina component/resource labels Aug 30, 2021

ci: add system tests to windows

1b2b440

deepankarm force-pushed the feat-windows-beta branch from 21cd98b to 9f3edf3 Compare September 1, 2021 11:44

deepankarm force-pushed the feat-windows-beta branch 6 times, most recently from 5ae4313 to 21f8926 Compare September 21, 2021 10:17

ci: nightly tests for windows

21601ed

deepankarm force-pushed the feat-windows-beta branch 2 times, most recently from 3fafb3b to c28432b Compare September 21, 2021 12:40

test(multimodal): skip on windows

5366867

deepankarm force-pushed the feat-windows-beta branch from c28432b to 5366867 Compare September 21, 2021 13:23

deepankarm changed the title ~~feat: windows support (alpha)~~ feat(windows): beta support Sep 21, 2021

deepankarm mentioned this pull request Sep 21, 2021

ci(docs): deploy docs from any branch #3456

Merged

deepankarm marked this pull request as ready for review September 21, 2021 15:32

deepankarm requested a review from alexcg1 as a code owner September 21, 2021 15:32

alaeddine-13 added 3 commits September 22, 2021 16:25

fix: seek body to end after opening mmap object

2266db5

chore: cleanup

de7a1c6

fix: linting

d4138d7

deepankarm commented Sep 23, 2021

View reviewed changes

alaeddine-13 added 2 commits September 23, 2021 07:50

chore: remove debugging

ecef990

Merge branch 'master' into feat-windows-beta

5075510

hanxiao reviewed Sep 25, 2021

View reviewed changes

hanxiao requested changes Sep 25, 2021

View reviewed changes

Merge branch 'master' into feat-windows-beta

806dcd7

hanxiao merged commit 2909933 into master Sep 26, 2021

hanxiao deleted the feat-windows-beta branch September 26, 2021 07:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(windows): beta support #3290

feat(windows): beta support #3290

deepankarm commented Aug 30, 2021 •

edited

Loading

codecov bot commented Aug 30, 2021 •

edited

Loading

github-actions bot commented Aug 30, 2021 •

edited

Loading

deepankarm Aug 30, 2021

deepankarm Sep 23, 2021

alaeddine-13 Sep 23, 2021

deepankarm Sep 23, 2021

alaeddine-13 Sep 23, 2021

hanxiao Sep 25, 2021 •

edited

Loading

deepankarm Sep 26, 2021

hanxiao Sep 26, 2021

hanxiao Sep 25, 2021

hanxiao Sep 25, 2021

hanxiao left a comment

github-actions bot commented Sep 26, 2021

feat(windows): beta support #3290

feat(windows): beta support #3290

Conversation

deepankarm commented Aug 30, 2021 • edited Loading

codecov bot commented Aug 30, 2021 • edited Loading

Codecov Report

github-actions bot commented Aug 30, 2021 • edited Loading

Latency summary

Breakdown

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hanxiao Sep 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hanxiao left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 26, 2021

deepankarm commented Aug 30, 2021 •

edited

Loading

codecov bot commented Aug 30, 2021 •

edited

Loading

github-actions bot commented Aug 30, 2021 •

edited

Loading

hanxiao Sep 25, 2021 •

edited

Loading