Merge pull request #58 from Kanaries/dev

refactor: state management -> mobx + rxjs

README.md

@@ -15,21 +15,26 @@ Augmented Analytics tool. Help you automate data analysis and visualization reco
 
 ## Introduction
 
-Rath helps you extract insights from datasource automatically and generate interactive visualization with interesting findings. Rath can generate dashboards from a raw dataset(as fact table) and explain some patterns to help you understand the data.
+Rath helps you extract insights from datasource automatically and generate interactive visualizations with interesting findings. Rath can auto generate high dimensional visualization contains complex patterns while most other auto-EDA tools only provideing simple low dimensional charts with basic statistics pattern. Its means you can use Rath to explore the data to a deep level and find more insights.
+
+Rath design a algorithm recommanding visualization with lowest perception error by human eyes, which means you can read the info in visualization much accuracte. 
 
 Here are main parts in Rath,
 
 ### DataSource
-dataSource board is for data uploading, sampling(currently support stream data, which means there is no limit of the size of file you uploaded), cleaning and defining fields type(dimensions, measures). In visual insights, we regard dimensions as independent variable or feature and measures as dependent variable or target.
+DataSource board is for data uploading, sampling(currently support stream data, which means there is no limit of the size of file you uploaded), cleaning and defining fields type(dimensions, measures). In visual insights, we regard dimensions as independent variable or feature and measures as dependent variable or target.
 
 ### Notebook
 Notebook is a board for user to know what happened in the automatic analysis process and how rath uses visual-insights. It shows how decisions are made by the application and provide interactive interface to adjust some of the parameters and operators used by the algorithm. 
 
 ### Gallery
 Gallery displays parts of the visualization with interesting findings. In Gallery, you can find interesting visualizaiton and use association feature to find more related visualization. You can also search specific info in gallery. There are some settings here to adjust some of the visual elements in the chart.
 
+### Explainer
+Explainer uses serveral insight discoverary algorithm to detect what is the specific insight type is shown in a visualization recommanded. Explainer is an extension of B. Tang 's Top K insight paper.
+
 ### Dashboard
-automantic generate dashboard for you. rath will figure out a set of visulization of which contents are connected to each other and can be used to analysis a specific problem. 
+Generate interactive dashboards for your. Rath will figure out sets of visulizations of which contents are connected to each other and can be used to analysis a specific problem. 
 
 ## Examples
 
@@ -49,27 +54,26 @@ Details of the test result can be accessed [here](https://www.yuque.com/chenhao-
 - [MacOS](https://ch-resources.oss-cn-shanghai.aliyuncs.com/downloads/rath/Kanaries%20Rath-0.1.0.dmg)
 - [Windows](https://ch-resources.oss-cn-shanghai.aliyuncs.com/downloads/rath/Kanaries%20Rath-0.1.0-win.zip)
 
-### run locally
+### deploy
+
+Rath now runs all the computation tasks on webworker. If you are interested in a server version, check the older version or contact us.
+
 (dev)
 ```bash
 # under project root dir
-yarn workspace visual-insights build
+yarn workspace graphic-walker build
 
 yarn workspace frontend start
 
-yarn workspace backend dev
-
 # localhost:3000
 ```
 
 production mode
 ```bash
-yarn workspace visual-insights build
+yarn workspace graphic-walker build
 
 yarn workspace frontend build
 
-yarn workspace backend dev
-
 # server:8000
 ```
 
@@ -78,53 +82,8 @@ only use the algorithm package. (`/packages/visual-insights`) ![](https://img.sh
 npm i visual-insights --save`
 ```
 
-## How does it work
-The working process are visualized in notebook board in the application.  *** Main process of the algorithm is shown in the `notebook` board. *** Here shows how rath use visual-insights to make a analytic pipeline.
-
-![](https://chspace.oss-cn-hongkong.aliyuncs.com/visual-insights/rath-arc.png)
-
-### Univariate summary
-For the first step, rath analyze all the fields in the dataset independently. It gets the fields' distributions and calculate its entropy. Besides, it will define a semantic type (`quantitative`, `ordinal`, `temporal`, `nominal`) for each field. More details of the field will be displayed when hover your mouse on the fields.
-
-![](https://cdn.nlark.com/yuque/0/2019/jpeg/171008/1570614609678-33d5f2c1-e51e-4bcd-8343-271a041f7519.jpeg)
-
-Then, it will find the fields with high entropy and try to reduce it by grouping the field (for example). Only dimensions participates this process.
-
-### Subspaces
-In this step, visual insights search the combination of fields. Visual-Insights suppose that any two fields appears in a view should be correlated with each other otherwise they should be display in seperated view. Visual-Insight now use crammver'V and pearson' cc for different types of fields' correlation.
-
-![](https://chspace.oss-cn-hongkong.aliyuncs.com/visual-insights/subspaces.svg)
-
-#### Correlation
-
-for example, the correlation of measures:
-
-![](https://chspace.oss-cn-hongkong.aliyuncs.com/visual-insights/correlation.svg)
-
-#### Clustering
-It helps you to cluster all the measures based on their correlation. It puts all the variables who are strongly related together to make a specific view (with specified dimenions).
-
-
-![](https://chspace.oss-cn-hongkong.aliyuncs.com/visual-insights/clustering.svg)
-
-### Insight Extraction
-After we get many subspaces, we can check the insight significance of each space. Currently, visual-insights support trend, outlier, group(whether different groups of data behave differently for spefic measures)
-
-![](https://chspace.oss-cn-hongkong.aliyuncs.com/visual-insights/rath-demo.jpg)
-
-### Specification & Visualization
-
-specification
-
-![](https://cdn.nlark.com/yuque/0/2019/png/171008/1570615741670-48941c9a-2788-4277-a946-6a75c400870d.png)
-
-visualization.
-
-![](https://cdn.nlark.com/yuque/0/2019/svg/171008/1570614529099-de4ead0d-5332-40c4-8101-e122ee0cf1d2.svg)
-
-
 ## Documentation
-
++ [Tutorial: Using Rath to find deep insight in your data](https://www.yuque.com/docs/share/3f32e044-3530-4ebe-9b01-287bfbdb7ce0?#)
 + visual insight api: [visual-insights](https://github.com/Kanaries/visual-insights/blob/master/README.md)
 + doc for reuseable hooks: todos
 
@@ -137,6 +96,3 @@ Rath is insipired by several excellent works below:
 + Vega-Lite: A Grammar of Interactive Graphics. Arvind Satyanarayan, Dominik Moritz, Kanit Wongsuphasawat, Jeffrey Heer. IEEE Trans. Visualization & Comp. Graphics (Proc. InfoVis), 2017
 + Cleveland, W., & McGill, R. (1984). Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods. Journal of the American Statistical Association, 79(387), 531-554. doi:10.2307/2288400
 
-## Story behind Rath
-
-The word Rath is used in SAO Alicization as name of org. developing Soul Translator (STL) and Under World, which creates A.L.I.C.E. ('Rath' is original from Mome Raths in *Alice's Adventures in Wonderland*.) I use the Rath as the name of the project for hoping it can create something far more than I could imagine like Alice.

README.zh-CN.md

@@ -8,14 +8,16 @@
 
 <img src="https://ch-rath.oss-ap-northeast-1.aliyuncs.com/assets/kanaries-light-bg.png" alt="logo" width="360px" style="" />
 
-Rath是新一代的基于增强分析技术的可视化分析工具，它提供了自动化的数据探索分析能力与自动可视化设计能力。Rath既可以在你对数据无从下手时提供分析入口的建议，也可以在你的分析过程中提供实时的分析辅助和建议。Rath会帮你完成大部分数据探索分析的工作，使得你可以专注于领域问题本身。
+Rath是新一代的基于增强分析技术的可视化分析工具，它提供了自动化的数据探索分析能力与自动可视化设计能力。Rath既可以在你对数据无从下手时提供分析入口的建议，也可以在你的分析过程中提供实时的分析辅助和建议。Rath会帮你完成大部分数据探索分析的工作，使得你可以专注于领域问题本身。当数据越复杂时，Rath的自动化能力带来的优势就越加显著。
 
 + [Youtube 视频 Demo](https://www.youtube.com/watch?v=o3_PH1Cbql4)
 + [Bilibili 视频 Demo](https://www.bilibili.com/video/av82089992/)
 
 ## Introduction
 
-Rath 可能帮助你快速完成对一个数据集的自动化可视化分析。它既可以帮助你在自助分析时提供实时的建议，也可以直接根据数据实时的特征生成数据报表。
+Rath 可能帮助你快速完成对一个数据集的自动化可视化分析。它既可以帮助你在自助分析时提供实时的建议，也可以直接根据数据实时的特征生成数据报表。与其他的一些自动化数据探索分析工具只能推荐简单的低维统计特征不同，Rath可以推荐包含一些深层规律的高维可视化，这使得你可以深度的探索数据。
+
+Rath 可以根据人眼的视觉感知的准确度进行可视化设计，这使得Rath设计的可视化可以被更准确的理解。这是基于感知学领域的实验数据设计的最优化算法，而不是一些不可量化的设计经验。
 
 相比于tableau、Congos Analytics等可视化分析工具，Rath可以大幅降低数据分析的门槛，使用户可以关注于实际的问题。
 
@@ -49,8 +51,9 @@ Rath 可能帮助你快速完成对一个数据集的自动化可视化分析。
 
 
 
-## Documentation
+## 资料
 
++ [《Tutorial: 使用Rath快速获取数据洞察》](https://www.yuque.com/docs/share/3f32e044-3530-4ebe-9b01-287bfbdb7ce0?#)
 + 使用Rath底层的SDK将增强分析能力嵌入你自己的应用: [visual-insights](https://github.com/Kanaries/visual-insights/blob/master/README.md)
 
 ## Reference

packages/frontend/package.json

@@ -5,7 +5,8 @@
   "homepage": "./",
   "prettier": {
     "tabWidth": 4,
-    "printWidth": 140
+    "printWidth": 140,
+    "singleQuote": true
   },
   "dependencies": {
     "@kanaries/web-data-loader": "0.1.0",
@@ -24,6 +25,9 @@
     "datalib": "^1.9.2",
     "fuse.js": "^3.4.6",
     "immer": "^4.0.1",
+    "mobx": "^6.3.2",
+    "mobx-react-lite": "^3.2.0",
+    "mobx-utils": "^6.0.4",
     "mocha": "^6.2.0",
     "office-ui-fabric-core": "^11.0.0",
     "office-ui-fabric-react": "^7.155.3",
@@ -35,12 +39,15 @@
     "react-json-view": "^1.19.1",
     "react-markdown": "^5.0.3",
     "react-scripts": "4.0.1",
+    "rxjs": "6.x",
+    "graphic-walker": "^0.0.1",
     "styled-components": "^5.2.1",
+    "symbol-observable": "^4.0.0",
     "typescript": "^4.1.3",
     "vega": "^5.19.1",
     "vega-embed": "^6.15.1",
     "vega-lite": "^4.17.0",
-    "visual-insights": "^0.5.5",
+    "visual-insights": "^0.5.6",
     "web-vitals": "^0.2.4",
     "worker-loader": "^3.0.7",
     "yarn": "^1.19.0"

packages/frontend/public/locales/en-US.json

@@ -18,6 +18,7 @@
     "dataSource": {
         "extractInsight": "Insight Analysis",
         "cleanMethod": "Clean Method",
+        "useField": "use field",
         "importData": {
             "buttonName": "Import Data",
             "type": {
@@ -99,7 +100,8 @@
         "summary": "Summary",
         "related": {
             "title": "Related Views"
-        }
+        },
+        "like": "like"
     },
     "dashBoard": {
         "generateButton": "Generate Dashboard",

packages/frontend/public/locales/zh-CN.json

@@ -18,6 +18,7 @@
     "dataSource": {
         "extractInsight": "开始分析",
         "cleanMethod": "清洗方法",
+        "useField": "允许使用",
         "importData": {
             "buttonName": "导入数据",
             "type": {
@@ -99,7 +100,8 @@
         "summary": "解读",
         "related": {
             "title": "联想视图"
-        }
+        },
+        "like": "喜欢这张图表"
     },
     "dashBoard": {
         "generateButton": "生成报表",

packages/frontend/src/App.tsx

@@ -1,30 +1,24 @@
-import React, { useMemo } from "react";
+import React from "react";
 import intl from 'react-intl-universal';
-import { useGlobalState, GlobalStateProvider } from "./state";
+import { useGlobalStore, StoreWrapper } from './store/index'
 import { Pivot, PivotItem } from "office-ui-fabric-react";
 import { useComposeState } from "./utils/index";
 import "./App.css";
-import RathLogo from './assets/kanaries-lite.png';
-import RathCoolLogo from './assets/rath-glasses.png';
 
 import Gallery from "./pages/gallery/index";
 import NoteBook from "./pages/notebook/index";
-import VisualEditor from './pages/visualEditor';
+import VisualInterface from './pages/visualInterface';
 import DataSourceBoard from "./pages/dataSource/index";
 import DashBoardPage from './pages/dashBoard/index';
 import DevPage from './pages/dev';
 import SupportPage from './pages/support/index';
 import UserSettings from './components/userSettings';
+import { observer } from "mobx-react-lite";
 
+// FIXME: 这两代码好像没什么用
 require('intl/locale-data/jsonp/en.js')
 require('intl/locale-data/jsonp/zh.js')
 
-const getLogoSrc = (withGlasses: boolean) => {
-  return withGlasses
-    ? RathCoolLogo
-    : RathLogo;
-};
-
 interface PageStatus {
   show: {
     insightBoard: boolean;
@@ -38,20 +32,21 @@ interface PageStatus {
 }
 
 function App() {
-  const [state, ] = useGlobalState();
-  const pivotList = useMemo(() => {
-    return [
-      intl.get('menu.dataSource'),
-      intl.get('menu.noteBook'),
-      intl.get('menu.explore'),
-      intl.get('menu.dashBoard'),
-      intl.get('menu.explainer'),
-      intl.get('menu.editor'),
-      intl.get('menu.support')
-    ].map((page, index) => {
-      return { title: page, itemKey: 'pivot-' + (index + 1) }
-    })
-  }, [state.lang])
+  const { langStore } = useGlobalStore()
+
+  let pivotKeys: string[] = ['dataSource', 'noteBook', 'explore', 'dashBoard', 'explainer', 'editor', 'support'];
+
+  let pivotList = pivotKeys.map((page, index) => {
+    return { title: page, itemKey: 'pivot-' + (index + 1) }
+  })
+
+  if (langStore.loaded) {
+    pivotList = pivotKeys.map(p => intl.get(`menu.${p}`))
+      .map((page, index) => {
+        return { title: page, itemKey: 'pivot-' + (index + 1) }
+      })
+  }
+
   const [pageStatus, setPageStatus] = useComposeState<PageStatus>({
     show: {
       insightBoard: false,
@@ -63,19 +58,24 @@ function App() {
       pivotKey: pivotList[0].itemKey,
     },
   })
+
+  if (!langStore.loaded) {
+    return <div></div>
+  }
+
   return (
     <div>
       <div className="header-bar">
         <div className="ms-Grid-row" dir="ltr">
           <div className="ms-Grid-col ms-sm6 ms-md4 ms-lg1">
             <a
               // onClick={() => { window.location.reload(false); }}
-              href="https://github.com/ObservedObserver/visual-insights"
+              href="https://github.com/Kanaries/Rath"
               className="logo"
             >
               <img
-                style={!state.beCool ? { width: '48px', marginTop: '4px' } : undefined}
-                src={getLogoSrc(state.beCool)}
+                style={{ width: '38px', marginTop: '4px' }}
+                src="/assets/kanaries-lite.png"
                 alt="rath"
               />
             </a>
@@ -105,7 +105,7 @@ function App() {
         </div>
       </div>
       {pageStatus.current.pivotKey === 'pivot-3' && (
-        <Gallery subspaceList={state.subspaceList} dataSource={state.cookedDataSource} summary={state.summary} />
+        <Gallery />
       )}
       {pageStatus.current.pivotKey === 'pivot-1' && (
         <DataSourceBoard
@@ -120,22 +120,24 @@ function App() {
       {pageStatus.current.pivotKey === 'pivot-2' && (
         <div className="content-container">
           <div className="card">
-            <NoteBook summary={state.summary} subspaceList={state.subspaceList} dataSource={state.cookedDataSource} />
+            <NoteBook />
           </div>
         </div>
       )}
       {pageStatus.current.pivotKey === 'pivot-4' && <DashBoardPage />}
       {pageStatus.current.pivotKey === 'pivot-5' && <DevPage />}
-      {pageStatus.current.pivotKey === 'pivot-6' && <VisualEditor dataSource={state.cookedDataSource} dimensions={state.cookedDimensions} measures={state.cookedMeasures} />}
+      {pageStatus.current.pivotKey === 'pivot-6' && <VisualInterface />}
       {pageStatus.current.pivotKey === 'pivot-7' && <SupportPage />}
     </div>
   )
 }
 
+const OBApp = observer(App);
+
 export default function WrappedApp() {
   return (
-    <GlobalStateProvider>
-      <App />
-    </GlobalStateProvider>
+    <StoreWrapper>
+        <OBApp />
+    </StoreWrapper>
   );
 }

packages/frontend/src/actions.ts

@@ -10,6 +10,7 @@ import {
   clusterMeasures
 } from "./service";
 import { GlobalState, StateUpdater } from './state';
+import { getGlobalStore } from "./store";
 
 
 
@@ -199,9 +200,8 @@ const getViewSpaces: Action<GetViewSpacesProps> = async (select, updateState, pa
 
 const extractInsights: Action<{dataSource: DataSource; fields: BIField[]}> = async (state, updateState, params) => {
   const { dataSource, fields } = params;
-  updateState(draft => {
-    draft.loading.gallery = true
-  })
+  const { galleryStore } = getGlobalStore();
+  galleryStore.loading = true;
   try {
     const univariateResult = await univariateSummary(state, updateState, {
       dataSource, fields
@@ -219,10 +219,7 @@ const extractInsights: Action<{dataSource: DataSource; fields: BIField[]}> = asy
       }
   } catch (error) {
   } finally {
-    updateState(draft => {
-      draft.loading.gallery = false
-      draft.loading.gallery = false
-    })
+    galleryStore.loading = false;
   }
 }