Skip to content

Latest commit

 

History

History
1115 lines (1090 loc) · 77.5 KB

planning.md

File metadata and controls

1115 lines (1090 loc) · 77.5 KB

Development feature planning

Upcoming releases

Targets for 1.6 (Apr 2014) - Theme: MLJS Ease of Use in the Browser

  • DONE BUG mljs.returnFacets returns true instead of this, breaking chaining
  • Widgets: Workplace completion
  • Basics
  • DONE drag/drop reordering of widgets working - DONE Re-order existing ones on to each other's DZ
    • DONE Reverse change of order doesn't move the DZ too - DONE BUG Not allowing existing last widget to be dropped on last widget dropzone, also handle not last existing re-ordering - DONE BUG dragging new class mid zone now breaks -> places new widget in last element of array, doesn't reshuffle undefined elements - DONE BUG multiple drops of new widgets on to end of same zone causes errors due to blanks between elements - added self._eliminateBlanks(z); call at start and end
  • Save function working - IN PROGRESS workplacecontext.prototype.save
    • TEST Get assignments from layout - self.generateAssignments()
    • DONE prune assignments (remove admin widgets)
    • TEST call mljs.saveWorkplace function - (this.container + "-save").onclick - show message that save was successful in error/notification area - ensure configwrapper event handlers are running even when moved around in HTML (via drag/drop)
  • findWorkplace working - TEST need to add valueConstraint support to options builder - need to add value query to query builder - move *Workplace functions from MLJS in to Workplace Context (they use structured search)
  • Context configuration
  • Enable all widgets for Workplace
  • remove widget button
  • multiple layouts working
  • Page load actions
  • Icons for widgets
  • Widgets: Navigation widget
  • One tab per Workplace in current app, sends to appropriate URL
  • Share workplace via permissions
  • Share workplace via URL - Add this to My Workplaces option on non-owned workplace page load
  • Edit limited to owner of workplace
  • DEFERRED Allow workplaces to be kept instead in a drop down menu
  • DEFERRED Menu Editor - DEFERRED Also allow fixed tabs at start and end (list for each). E.g. Settings, Home, etc. for non-workplace dynamic pages - display text, url
  • New workplace page button on end of nav widget
  • Powered by MarkLogic image (license?)
  • Practicality updates
  • Options builder UI widget - read from ML if possible
  • Query builder UI widget (base query for search, default query for page load)
  • Docs: OOTB Workplace demo app - no data
  • Widgets: UI Consistency
  • Widgets: Update all for Bootstrap CSS support
  • Docs: Update MLJSREST app framework for bootstrap support
  • Docs: Get hold of MarkLogic bootstrap theme CSS from MY (License?)
  • Get MY to review app, workplace and widget UX
  • Docs: Deprecate mldbwebtest app as primary source of widget files in build, remove app itself
  • DONE BUG Widgets: searchbar - When contributing query as term query for structured search, if text blank need to send and-query not term-query("") - else nothing is returned!
  • DONE BUG Core: version being called multiple times per page (when version endpoint doesn't exist)
  • IN PROGRESS BUG Docs: Explorer widget not loading document facet data on mljsrest app
  • DONE Due to subjectIri to docuri hack line 508 of widget-explore.js: Need better way to determine document URI from subjectIri -> should use #uri child property
  • Need to also cache links between docuri (key) and subject URI, so documentcontext listener can match subject correctly
  • Docs: Put latest doc updates front and centre of MLJS main page and API home page
  • aka release notes
  • youtube videos
  • blog (news) items
  • latest tutorials
  • Docs: Image previews of all widgets, linking to pages with code examples
  • DEFERRED Workplace page configuration JSON download/upload too?
  • DEFERRED Sample content for testing bundle too? (MLCP archive? Version dependent?)
  • DEFERRED Workplace page search and share app (hosting?)
  • DEFERRED Workplace widget NPM like search and install app (hosting?)

Targets for 1.8 (May 2014) - Theme: Visualisations

  • BUG Widgets: combined query - test against latest MarkLogic 7 build
  • BUG Widgets: Address Lookup: Google geocoder needs to fail gracefully if google libraries not available (offline mode)
  • Core: Search Context: Search context chaining
  • Add one or more child contexts, with a name for the contributed query term setting (E.g. parent contributes "collection:EmploymentStats AND year GT 2008 and year LT 2013")
  • Ensure each context may have a hidden query term in it (E.g. one child has extra of Year=2010, one child has extra of Year=2011, and so on)
  • Test this working with search page affecting master search context (and query), highcharts depending upon child search context
  • DEFERRED detect circular relationships (infinite JavaScript loops)
  • IN PROGRESS Widgets: Search Results
  • IN PROGRESS Improve support for pluggable rendering for areas of result pane:-
  • IN PROGRESS title
  • selection tickbox
  • IN PROGRESS actions (pluggable) - show more detail (expanded result summary) - show less detail (default short result summary) - show similar (text or thumbnail) - show related/linked subjects/documents (navigable diagram with home button, embedded) - popup preview (popup pain) - popup delete - go edit (goto page - edit) - go all details (goto page - view) - go explore (goto page - semantic data explorer) - DONE new-window view (download - /v1/documents) - DONE new-window view (pretty xml - /v1/documents with transform)
  • similar documents (text or thumbnail) under actions bar, hidden by default
  • related documents (text or thumbnail) under actions bar, hidden by default
  • related facts (semantic triples)
  • thumbnail(s) on right hand side with navigation if multiple - DEFERRED preview to occur on hover of thumbnail, if supported - thumbnail caption
  • latest comments (similar to related documents) - add comment support - ratings support in same area
  • DEFERRED expanded result summary (controlled by action or global show detailed results setting)
  • Ability to globally show/hide named sections
  • IN PROGRESS Make it easier to add event handlers like click to navigate, view (as html via XSLT), download
  • Allow passing of custom parameters to event handlers in addition to document uri
  • Externalise constraint name to display name (search option annotations on constraints) - constraintTitle function on manager
  • Core: data series context
  • Handles series calculation and parsing external to any particular widget
  • reliant upon a single searchcontext (and options builder) object
  • list potential data sources
  • list potential category x sources
  • list potential value y (measure) sources
  • calculate series and their data points dynamically when results arrive, based on above configuration
  • fire series update event for individual series (need depends upon widget)
  • fire all series update event for overall results (need depends upon widget)
  • Widgets: Workplace
  • Allow user personalisation for widgets on a page (auto save and load via workplace context)
  • Allow showing of live data against configuration options (E.g. list the facets from options that are selectable for a highchart)
  • Widgets: Highcharts
  • editable flag on chart - uses widget personalisation
  • Select chart type dynamically
  • Multiple source support
  • Facet totals
  • Co-occurence values - THIS is likely to be the most common use case. E.g. Category x is actor, value y is quantity, series is Genre (actor-genre co-occurrence)
  • Values call - UDFs etc - TBD
  • DEFERRED in document values (requires pulling all doc values, or just extracted meta, to browser for multiple results pages)
  • Filtering support
  • Select category x
  • Range for continuous x (e.g. date)
  • DEFERRED filter affects overall search query across all widgets
  • Quick altering - aka Pivot Charts
  • select value y source
  • select dynamic series source
  • Docs: Create mock sample search pages
  • wordpress like demo summary view for documents
  • google like summary, with similar stories indented
  • tabular view - http://www.webdesignerdepot.com/2010/02/showcase-of-great-web-app-interfaces/
  • google images like summary view, and detail on click
  • Widgets: Sparqlbar: Allow relatedFrom relationships as well as current relatedTo
  • Widgets: Explorer: Save URI to/from IRI mapping prior to calling document facets fetch (currently based on name substring - dangerous)
  • DEFERRED Widgets: Search Results: Allow rendering with other layouts rather than just a simple column
  • Widgets: 2 column across-first cell rendering layout
  • Widgets: 3 column across-first cell rendering layout
  • Widgets: zoomable n row rendering layout (like on springer images website)
  • DEFERRED Add support for result drag and drop to other areas
  • Widgets: Search Results: Default Rendering improvements
  • hierarchical XML viewing (down three levels?)
  • IN PROGRESS triple support (with summary?)
  • Docs: Update sample apps to make best use of search presentation features
  • Widgets: OpenLayers
  • Support multiple result types (series) from same searchcontext. E.g. matcher() and series() functions (use same pattern as searchresults for simplicity)
  • Missed 1.2 (Dec 2013) - High Priority - targets
  • IN PROGRESS Widgets: Data Explorer improvements
  • DONE: Widgets: Data explorer widget shows "Loading..." when attempting to draw a link to an entity that already exists in the visible graph. Should just draw an arrow.
  • ? Widgets: Data explorer widget: Need to ensure arrows are drawn behind boxes z-order
  • ? Widgets: Data explorer widget: Need to set end points for existing boxes to edge most in line with startpoint to centre of box, but finishing outside the box
  • DONE BUG: Widgets: Data explorer widget shows extra two boxes when drawing Wendy (Doesn't remove original link boxes when redrawing?)
  • ? Widgets: Give the data explorer the same document title heuristics as the search results widget. (E.g. XHTML title element value)
  • Widgets: Add a view document link to the data explorer when rendering a document node (Links to different app URL)
  • Immediate explorer
  • Left hand side Subject of interest
  • Related content below
  • Related subjects to right
  • Navigating onward redraws, shows parent (previous) Subject above new subject of interest

DEFERRED from 1.8 Theme: Visualisations

  • Widgets: Document Collector dropbox (also supports the searchresults document title, result summary plugins)
  • Widgets: search bar: Suppress facets, sorting shown within text query bar
  • Widgets: search bar: Support boolean AND, OR, ( and ), NOT
  • Widgets: save search
  • Widgets: searchsorter: OOTB support for newest first/oldest first based on MarkLogic created and last modified dates
  • Widgets: searchperpage: Number of results to show per page 25 | 50 | 100 for example
  • Widgets: layouts to support collapsable zones (like springer images search modification left hand column)
  • Widgets: MLJS App wide settings in DB, generated from /mljs/app.json on webapp directory (E.g. for new application)
  • Core: searchcontext
  • Rework how facets are contributed - can't override at the moment, only contribute - causes facets to reappear
  • Allow AND or OR queries for text queries by default (defaults to AND at the moment)
  • Support for non exact matches (query options? query itself?) - E.g. wildcards
  • Missed 1.2 (Dec 2013) - Medium Priority - targets
  • IN PROGRESS Widget: RDB2RDF in test app
  • Remove jQuery specific code
  • DONE Check Roxy installation of REST extension works automatically
  • DONE validate CSS works correctly on all pages
  • Missed 1.2 (Dec 2013) - Low Priority - targets
  • App builder gap closing
  • Widgets: searchbar - Auto complete (suggestions), including constraint name suggestions
  • Widgets: HighCharts - Date time facet selection (column charts like markmail - allow vertical support (bar chart) too for better granularity)
  • Widgets: searchfacets - Split out listing/altering selected facets (as opposed to facet value list) in to own widget. E.g. to be used with chart/co-occurence pages with a f facet list widget
  • DEFERRED Shadow queries (MLJS' alternative mechanism)
  • DEFERRED Simple/complex view of results (AppBuilder achieves this via state configuration in search options)
  • Widgets: OpenLayers - Use MarkLogic as a tile cache rather than the browser (and do "cache first")
  • NOT A BUG searchfacets or searchbar widgets using facet title, not constraint name, to indicate constraint added to searchbar (also for lookup/matching)
  • NA Actually, there is no 'facet title' just that constraint name and field name are generally the same. Need to add Title support somehow (DONE via annotation[0])
  • New 1.4 (Feb 2014) targets:-
  • Widgets: New Visualisations (in order of priority)
  • Widgets: Folder browsing widget (Probably easiest to do like Mac child folders, left to right, rather than a tree structure)
  • Widgets: Hive diagram - http://www.hiveplot.net
  • Widgets: Interactive data exploration widgets (E.g. pivot chart) - Widgets: Drag & Drop semantic context object browser with support for menu popups 'add as series, add as secondary axis' etc.
  • NA Widgets: Use highcharts explore instead - Widgets: InfoVis network diagram
  • Widgets: Node diagram. Show images, other metadata. Force directed using selectable methods. Support for clustering. Navigable (not all drawn at once)
  • Widgets: XML/JSON document creation-by-example (via HTML form generator)
  • Widgets: Alerting client widget (Requires Alerting API, CORS) with popups - Core: Node.js Alerting middleware support
  • Widgets: 2 way Co-occurence drawn as x by y grid, with size and colour of points relating to values
  • Widgets: Setting up system alerts (NOT V7 REST alerts) to send new docs to specified URL (as PUT or POST requests, configurable)
  • Widgets: Interactive query builder - Document type to retrieve - Geospatial co-ordinate selection - Lookup existing indexes - Specify query option constraints - Create constraint setting interactively - Save this to named options document on server
  • Widgets: Sparql + content search + values results download - Download CSV file of results - Download generated PNG from current visualisation div
  • Widgets: Situational Awareness: Ensure OpenLayers map supports features of existing SVG map viewer. If not then create SVG based map based on existing demo - Shows live map tiles feeds. Support for multiple tile layers overlapping - Vector drawing of information - Rotation, translation (lateral movement) of map - Edge of screen flashing alerts for new information of importance just out of view - Command letter shortcuts (scroll, zoom, show/hide features) - Mini map to show activity just out of view (Like the game Z)
  • Widgets: Existing widget improvements
  • Widgets: HighCharts: Support multiple series, axes and searchcontexts (how?) -> Use work in OpenLayers as a base
  • Widgets: HighCharts: Support 'diving down' within a result set in a single chart. E.g. different 'series' settings and queries for different 'zoom' levels - Use Case 1: Time series - zoom to year, zoom to month, zoom to day - wouldn't require a new query or different series configurations - Use Case 2: Deep dive - Clicking on Sales Region vs. Yearly takings shows individuals vs. yearly takings. Clicking again shows an individual's accounts and sales stage. Will require new queries and series at each level (probably a new searchcontext per level) - Back Up button - Reset button
  • Widgets: HighCharts: Charts widget to support extraction of XML info via XPath
  • Widgets: HighCharts: Support array children in aggregations for graphs
  • Widgets: Highcharts - updateSubjectFacts support (extracting series from ?s ?p ?o and ?g in sparql results) - updateSparqlResults support (for any arbitrary named sparql binded variables, not just ?s, ?p, ?o and ?g) (pull back known predicates in semantic config by ?name) - Ensure charts are efficient when rendering facet and in doc values, and triples
  • Widgets: searchresults: Adding action bar support, with plugins for each button (applicable() and render() functions) - E.g. Download, View, Generate PDF, Delete, Properties
  • Widgets: searchfacets: Support facet hierarchies
  • Widgets: searchfacets: Facet support for buckets, sliders
  • Widgets: searchfacets: Facet hierarchies in browser sidebar
  • Widgets: searchbar: Add search action button - 'save as alert'
  • Widgets: sparqlbar: Other sparql improvements (via sparql builder) - Support OPTIONAL? - Support UNION? - Sparql query builder in mljs core? - IN PROGRESS Support FILTER?
    • DONE Used this for < > <= >= != for integer, double fields
    • Regex? (used by suggestions only at the moment)
    • TODO validate string comparison instead of numeric comparison as reported by Ken Tune in SYTYCD demo - Support REDUCED? (Permits duplicates to be eliminated) - Support FROM NAMED (limit results from set of named graphs) - http://www.w3.org/TR/2008/REC-rdf-sparql-query-20080115/#restrictByLabel - NA multiple listeners for this in same context - Multiple modes, one to contribute sparql query to a searchcontext, another to perform sparql query itself (to list entities) - NA Sparql 1.1 required - Allow less than / greater than rather than just equals, depending on property selected - Aggregations of triples for Charts to work alongside facets of documents
  • Widgets: Kratu: Support array children in Kratu
  • Widgets: Kratu: Support CSV in Kratu
  • Widgets: Workplace: W3C CORS: Support for using widgets in non ML 6 REST webapps, and embedding widgets remotely (i.e. secure cross site scripting support)
  • Widgets: Workplace: Moveable column layout handler
  • Widgets: Workplace: Style overrides. Hardcoded CSS for this version. Useful for layouts in particular. Also at page level.
  • Document all new widgets
  • Tests for new widgets

Floating Targets

  • CSV Import improvements / testing
  • Checkout XSLT upload and transform method
  • Test PAF file format
  • Nice to have: Widgets: Browse related Subjects in node diagram
  • TEST Core: Add support for save/get/delete/list transforms
  • LOW Widgets: Transforms administration
  • LOW Upload (New) transform / Edit / View
  • LOW List transforms
  • LOW Add test document + run XSLT
  • LOW Advanced Edit - syntax highlighting for XQuery/XSLT
  • LOW Widgets: Information flow administration
  • LOW List pre/post commit triggers, action module name, whether acting on collection or URI base
  • LOW List alerts, match criteria, actions list
  • LOW List CPF pipeline information and flow
  • LOW List norm configurations and denormalisation targets (including XSLT invocation)
  • SMA
  • High:-
  • Widgets: Advanced search editing - Support for boolean logic, word and phrase matches, NEAR searches, constraint drop downs - DONE Basic named constraint mode (AND'ed text query of all, contributed as facets) - Parsing existing text query - Add/remove clause buttons - Clause types: Boolean: AND(0+), OR(0+), NEAR(2) with number, NOT(0+) - Clause types: Basic: Word or Phrase, Constraint Value (drop down constraint, lookup or type value), Thesaurus (Custom) value - Clause types: Advanced: Range Constraint (min, max, lt, lte, gt, gte), Range fixed buckets (from options), range dynamic (time) buckets (last 2 days) - Low: Clause types: Structured: geo box/radius/polygon
    • Drop down for 'areas' (polygons pre-defined)
    • Widgets: List saved search areas, new/edit/delete links
    • Widgets: OpenLayers map used to define polygon - 'save polygon' link with name
    • Saves data as a search query in JSON to a document with a wrapper, including area display name and type display name (constituency, country, state, AoR etc)
  • Widgets: Save last executed search
  • Widgets: Load saved search drop down (Last 10 saved, link to full list)
  • Widgets: OpenLayers - show detail on hover, perform full action on click (E.g. preview appears mid page)
  • Research: Most mentions as a sorting factor? Can we index this number using a path constraint? //tw:comments/fn:count(.) ?
  • Widgets: docview: common Social Media formats support - Twitter, Facebook (messages and profile pages), with links to viewing online on original websites (or embed)
  • Research: How to get messages in there - task server spawning task per capture method, option for global enable/disable on collectors
  • Widgets: Collector list, links for new, delete, edit, task server checking schedule - REST extension for task server set up
  • Widgets: New/Edit collector - Twitter Tweets - DEFERRED Twitter Profiles - DEFERRED Facebook messages - DEFERRED Facebook profiles
  • Medium:-
  • Widgets: List my saved searches
  • Widgets: alert set up - Widgets: New/Edit alert - name, action type, action settings, pre or post commit - Alert Action: Tag match - element name(country), namespace (opt), word query ("UK" OR "United Kingdom") match attributeNameOpt (@country), attribute value("UK") - Widgets: List saved alerts (including type of action) - REST extension to save/modify/list alerts - Widgets: Preview alert search matches in edit mode
  • Widgets: Workplace Layout: Full screen width - Widgets: OpenLayers test auto using full screen width, config option on child layouts to include container class name(s)
  • Alert Action: Gazetteer mentions_location support for Entity Extraction tagging and Enrichment (lat,lon)
  • Research: Plain text location to lon/lat accuracy tagging (ask AT)
  • Config: Twitter, Facebook specific location geotagged field support (as a field constraint across multiple sources)
  • Alert: Within polygon (county, country (within UK), parliamentary boundary) for Entity Extraction tagging
  • Low:-
  • Widgets: Enable search as alert, with drop down for actions (Email, IM, etc.)
  • Widgets: Workplace app level settings. - Specifically, default role to see / edit shared searches - Default REST Server search options set up (constraints)
  • Refactor: Move widgets' use of 960.css out in to a Layout instance, as config options on the assignment itself (not widget or layout)

Targets for 1.9 (Jun 2014) Interim Release - Theme: Documentation

  • Missed 1.2 (Dec 2013) - Medium Priority - targets
  • Docs: Tutorials
  • (Emil Zegers) RDB2RDF quick start tutorial
  • (Emil Zegers) OpenLayers quick start tutorial
  • IN PROGRESS Docs: Further MLJS docs
  • Docs: Search results - parsing REST JSON for results
  • DONE 006-widget-page.js in Samples: Widgets: Page template js file - for those wanting to create a new MLJS powered page
  • IN PROGRESS Widgets: Search Speed - As a tutorial for those wishing to contribute to MLJS by creating new widgets of their own
  • Docs: documentcontext object
  • Docs: tripleconfig
  • Docs: semanticcontext
  • Figure out how to document own JSON formats within jsdoc and link to external references of them. (E.g. REST API)
  • Core: Document return types for all functions
  • Widgets: Document return types for all functions
  • Docs: Internal JSON formats used, and links to REST API ones exposed to end users (E.g. search results) - search formats - search options - JSON internal predicate format - JSON internal rdf type format - JSON external rdf type format - documentcontext.allowableProperty JSON - search properties transform parameters sprops_opt.transformParameters - sortOrder element/field/attribute spec
  • Docs: MLJS Webinar series of Video Tutorials
  1. DONE Introduction to MLJS (text done. 3min 15) - Why should I care? (Akin to the first 5 minutes of my company wide webinar)
  2. IN PROGRESS Introduction to the MLJS Core API (text done. 4min21). Core communication with MarkLogic server. Introduction to Search Context and other Context objects
  3. DONE Introduction to the MLJS Widget API. Briefly show the mldbwebtest app, deploying it, and a brief description of the widget test pages
  4. Creating a new Hybrid app with MLJS. (Already exists, on blog site). Creating a search page and configuring options and facets. A search page with two charts (akin to an AppBuilder demo) a. Creating a new REST app with MLJS
  5. Customising how search results are displayed - The Renderer architecture. Detect and display data in a search results page in new and interesting ways
  • Docs: MLJS Webinar series of Video Tutorials
  1. Loading documents with MLJS. Add test XML, JSON and text documents using MLJS and a test loading web page.
  2. Using MLJS to explore data - Integrated charts, co-occurence, map, query bar
  3. Loading triples with MLJS. How to use a simple JSON structure to load standalone facts.
  4. Configuring an ontology to power Semantic Queries with Sparql. How to customise a sparql query bar widget, and create a subject browsing page
  • New 1.5 (Mar 2014) Interim targets
  • CSS best practice
  • Use variables where it makes sense (title height, padding, etc.)
  • minimise property name length and collisions with other css by using .mljswidget .docproperties { .title {} .name {} ... } style of CSS declarations
  • Rejig H tags to be at a lower level (searchresults h5, subtitle h4, title h3) - leaves space for page title (h1) and panel/tab title (h2), and extra in widget title. E.g. search property value (h6)
  • IN PROGRESS create generic classes for all widgets - DONE title - subtitle - listtitle (E.g. for search results, sparql subjects, other dynamically drawn h4 type tags) - highlight - selected
  • JavaScript best practices / conventions
  • IN PROGRESS use var max = whatever.length in for loops - between 2 and 150 times faster than accessing the DOM on each iteration - DONE Sparql (triple) widgets - Search widgets - Core - Highcharts
  • underscores for private functions
  • NA _configuration json object for Workplace compatible widgets/contexts
  • UpperCamelCase for constructors/objects
  • IN PROGRESS lowerCamelCase for functions (searchcontext and mljs have some alllowercase methods)
  • DONE Using Factory pattern. See mljs.create*() methods. Changed mljs.defaultconnection in core MLJS to instead access parent MLJS instance
  • DONE Using logger mixin. Logger's attached to DB, so have had to link tripleconfig to creating db via factory method. keep tripleconfig separate from core MLJS instantiator (as it never requires a connection to use), but allow a way to specify a logger and check for this (to avoid defaultconnection.logger)
  • DONE better way to access logging functionality by default - define _i(), _d() prototype methods for each subobject? Via a core mixin utility func? - DONE (Could use same mechanism to provide correct 'this' reference from constructor function.) - DONE Using mljs.create*() factory methods. Have mljs.prototype. constructor functions add a db property pointer to all created children
  • NA Using factory pattern instead. Add check in constructor functions to ensure if not called via new MyObject(), that it returns a new instance anyway (location 1051)
  • Double check isArray function definition for best practice (location 1110) - also check for splice()
  • IN PROGRESS Done this for tripleconfig, not widgets and their contexts. Use self defining functions for lazy loading (E.g. semantic context creation), and any requiring extensive initialisation (Sparql widget?) (location 1515)
  • Memoization (cache) pattern applied to sparql facts (location 1702)
  • Apply Currying when calling the same function with mostly same parameters (if costly) (location 1860)
  • Use Module pattern and closures to hide internal properties/private methods (location 2117) but with constructors (2139)
  • DONE using for logging mixin on context objects. Apply Borrow and Bind pattern (mixin for functions) where applicable (to avoid any inheritance temptations) (location 2922)
  • Apply Data Validation pattern to 'create' widget (When new functionality added - may be V 1.2?) (location 3254)
  • Find a better way to restrict what parts of the logging output are set to use debug (Perhaps a mixin to objects with lazy initialisation based on debug level specified, if any)
  • Use more high order functions (callbacks, replacing functionality within otherwise identical loops)
  • DONE Check for 'innerHTML += ' and replace with appendChild and temporary div trick
  • Perform JavaScript profiling in FireBug (again) to check for issues
  • Auto install of extensions referenced by MLJS core via PUT /v1/ext endpoint
  • IN PROGRESS rdb2rdf
  • savedsearch
  • alerts
  • version detect (future proofing)
  • dls
  • whoami
  • Test pages for each of the above too
  • Core: Query Builder.
  • Easy geospatial search (Already done?)
  • triples-range-query support
  • Core: Support all search functions not currently provided in query builder
  • IN PROGRESS Core: Extra query builder options (all constraints supported by options, plus sorting)
  • DONE Term query
  • Word query
  • V7: result decorator option
  • V7: Boost query & matching query
  • Others?
  • Core: Complete support for /v1/values REST function (suggestions aka auto complete etc)
  • Test: Support current functionality against MarkLogic V7
  • Core: New V7 functionality
  • Core: Multipart/mime support - required for POST search, but also useful generically. May require some additional core work
  • Core: POST /v1/search - Dynamic search options (Try this in search context, catch error if on V6 and fall back to original mode)
  • Core: Alerting
  • Core: Score-function=linear|reciprocal, scale-factor=0|inputFloat
  • Core: Supply pagination values as part of query (as Vis Widgets do) not as part of search options
  • Core: Support all client API /v1/ REST endpoints in ML7
  • Node: More Node.js tests for all new functionality
  • Docs: Document new widgets with screen shots and scenario code
  • NodeJS: Support SSL
  • NodeJS: SSL with Basic
  • NodeJS: SSL with Digest
  • NodeJS: Basic + Digest non SSL
  • NodeJS: SSL Basic + Digest
  • NodeJS: SSL no auth (default user)
  • NodeJS: Support 'anyauth' option, much like curl (i.e. auth method auto detection)
  • Core: Search Options builder
  • hidetriples() function to hide any docs under /triplestore/ (NOT any that define sem:triples)
  • Missed 1.2 (Dec 2013) LOW priority targets
  • MarkLogic reported bugs retesting
  • BUG (Product 6.0-2): plain text documents - text content not escaped properly
  • Own demo apps
  • Update Situational Awareness Node.JS app for 0.8+ MLJS
  • Update alerting instant messaging Jabber Node.JS app for 0.8+ MLJS
  • IN PROGRESS REST extensions
  • DONE Core: DLS declare document as a record
  • Core: DLS list all documents in a record collection
  • DONE Core: DLS list all DLS collections
  • Core: DLS list all retention rules
  • Core: DLS get document history
  • Core: DLS get document version
  • Core: DLS add retention rule
  • Widgets: DLS add retention rule widget
  • DONE Widgets: DLS declare search results as record widget

Targets for 2.0 (Jul 2014) - Theme: Lifecycle

  • MarkLogic version 8 support completed
  • Widgets: HTML content editing widget (new doc, and existing document with specified element (E.g. body, comments) and configuration for what is permissable as content)
  • Widgets: Situational awareness and search renderer app (from BF sim)
  • Widgets: Table - allow binding of JSON search results to jQuery table (for example)
  • Widgets: Table - allow binding of XML search results to jQuery table (for example)
  • Widgets: Tree - browse JSON/XML documents in search results as a tree
  • Widgets: Timeline (point in time and start / stop dates)
  • Widgets: Rose - direction (N, S etc) and time (12 and 24 hour) - arbitrary categories
  • Widgets: File upload (multiple)
  • Widgets: (multi) Upload progress bar
  • Test: prototype.js browser AJAX binding

Targets for 2.1 (Aug 2014) Interim Release - Theme: Ease of migration to MLJS

  • MarkLogic version 8 fixes and additions
  • IN PROGRESS NodeJS: Node.js tests for all functionality now in core MLJS
  • DONE Convert all assert usage to assert = require('chai').assert (better exception handling)
  • DONE Once assert changed, modify tests to remove skipped tests
  • Add 'sunny day' success tests - at least one per function
  • Add tests for each bug previously squashed/reported to ENG
  • Docs: Further video tutorials
  1. Combining Document and Semantic search. Extend the semantic page to show 'related documents'. Create a data explorer page to show subject and embedded document information.
  2. Creating an application. Linking existing pages together with actions. Search for content, view document metadata, download the related PDF version.
  3. Customising the Chrome of your application. How to customise an entire MLJS based Roxy hybrid app for a particular client's web theme.
  4. Using MLJS with your favourite widget library. Using an AngularJS page controller and widgets integrated with MLJS powered widgets
  5. Writing your own widget I. Create a search metrics widget with a refresh button to re-execute the search.
  6. Writing your own widget II. Create a multiple document upload widget, with progress bar, named collection, and link to a search page to show those just uploaded.
  • App Builder to MLJS+Roxy automated conversion
  • Point at Roxy generated App Builder download / local file system copy of appbuilder folder
  • Copy over rest extensions as required
  • Support pure REST and hybrid modes for app generation
  • Generate standard "/" page and register as Workplace page JSON editable by admin account
  • Layout thinthick | thinthick1chart | thinthick2chart
  • Widget assignment and default config
  • Copy over search options (and thus facet info) - from the db?
  • Facet display names from JSON config file
  • REST request re-writer to point to Workplace page (mljs-workplace.html.xqy), pass on other requests to REST API
  • Document preview page (may apply XSLT) on /view?uri= - check the appbuilder URL for this
  • Roxy commands for creating entire MLJS powered pages, controllers, and template JavaScript (E.g. ./ml mljs mycontroller/search searchpage leftchart=pie rightchart=map options=mysearchoptions )
  • DONE Other cool things we could do with MarkLogic
  • DONE SYTYCD Combine triple search with content search - limit documents returned from triple store by term (word) query
  • Alerting with triples (+drawbacks - document level, not triple level)
  • Widgets: InfoBox based on common name of search via updateResults (or updateQuery?) - Matt W
  • Widgets: Potentially of interest. Spelling correction and/or thesaurus lookup of search query word(s), then use those for subject lookup - Matt W
  • PROV-O ontology support
  • hide by default, show provenance button to view provenance and interrelationship of entities
  • requires node diagram for viewing effectively (use PROV-O diagram shapes/colours? Over-Time view?)
  • may be easiest to import OWL2 ontology in to triple config
  • (Emil Zegers) Tools: Windows batch file versions of developer required (non build related) bash scripts, including mljsme
  • mljsme
  • TEST ml.bat roxy deploy
  • create-mldbtest
  • delete-mldbtest
  • upgraderoxy
  • Beyond App Builder
  • Action bar to do things with search results
  • DONE Multi page results selection
  • More HighCharts chart types in search results page
  • Multi axes in HighCharts
  • Multi series in HighCharts (not just split by category)
  • Multiple facets in HighCharts
  • HighCharts to support SPARQL facts results (aggregation workarounds?)
  • Save search for later use
  • Test of standalone REST app via Roxy in V7
  • Hover over search results fires searchcontext highlight action
  • Document context highlighting configurable to fire get semantic info on MarkLogic document
  • MarkLogic document ontology
  • Workplace context improvements
  • updatePreferences - used by widgets to globally (or in just this workplace page/sub layout) save and load preferences about this widget. E.g. last displayed place
  • loadPreferences
  • DEFERRED updateShareable - when executed as embedded shareable page, load all workplace page settings, and widget 'preferences' to use (as wont be 'current' user) - handle widget preferences transparently in this case - no special widget settings
  • DEFERRED Widgets: Share Workplace widget - button, and configuration to share (assume security is default app security for now)
  • IN PROGRESS Defines what pages are available, the layout of those pages, and what is shown on each
  • Alter mljsme.sh to copy in a Roxy controller and view using default configured layout - equiv of ml create mljs/main html followed by copying scripts over
  • Alternatively, build Roxy command extension to do this. Use mljsme.sh to integrate these in to a Roxy app
  • Load application wide Workplace configuration settings (Not just content DB wide - use domain name somehow)
  • IN PROGRESS Widget initialisation for each page done dynamically
  • NA (For first version require all to be on import path) Requires dynamic loading of scripts, or importing of all scripts (do latter for now)
  • Ability to link widgets together. E.g. dropping a searchbar creates a searchcontext - list which have endpoints to link to which contexts' functions - like Yahoo Pipes
  • By default, every page has a searchcontext and a semanticcontext, and all widgets by default linked to both, where applicable
  • DONE How to manage this linking dynamically in code?
  • DONE List contexts and map them to widgets
  • IN PROGRESS Handle widget configuration, and actions
  • DONE On page load actions (execute search - context methods)
  • IN PROGRESS Instance widget configuration settings (Easier than using 'call javascript' action) (hard coded in config as opposed to personalisations) - DONE search widgets - DONE high charts widgets - DONE searchcontext - semanticcontext - tripleconfig - documentcontext - cooccurence - kratu - upload - viewdoc properties - triples widgets
  • Workplace configuration widgets required
  • Separate panel for linking
  • Edit mode shows this panel (powered by widget configuration object)
  • Show chosen layout with which widgets assigned to which place
  • Show widget palette and support drag and drop on to layout zones (in main page?)
  • Save configuration after every edit
  • Reset to defaults option
  • Copy existing Workplace Page
  • Allow users to switch their Workplace on the fly - Widget: Workplace selection widget (drawn manually at top of each page, or as an option in Workplace widget)
  • Support Duplicating existing Workplace
  • OOTB Workplace examples
  • DONE MLDBWebTest Animal Watch Workplace
  • Semantic Workplace
  • Editable blank workplace sandbox page
  • Support pre-made templates
  • Some built in (Search page aka AppBuilder, semantic search page, data exploration page, document details page)
  • Custom made templates - Page 'save as template' mode in edit screen - allow users to 'save' their own templates against their account only - allow sharing of templates to specified roles in ML (just read permission setting)
  • Docs: Video tutorials for all new functionality - Create a Semantic Application Series
  • Intro to semantics in MarkLogic
  • Manually load in some triple in N-triples format, and query this in QConsole and rest to show the ideas behind what we're doing - Include a sprinkling of 'why do this with triples?' / 'why triples are cool!' type slides
  • Do a query that you can't do in relational / document search alone -> Idea of context, defining the 'relationship' not just the presence of data
  • Manually define a new triple (mention we'll suggest triples in the future) for a XHTML document
  • Includes setting up your own ontology in the semanticcontext object
  • Include 'automatically link root entity back to document' via documentcontext object (to be created)
  • Add sparql query widget using same ontology
  • Add entity facts and content linking search widget
  • Add relationship visualisation widget
  • Implying triples from document content
  • define a trigger to highlight key XML with known terms
  • link these xml elements to the ontology via the suggestions widget
  • Inferencing
  • Start with a query for which you have the data, but which creating a sparql query is non trivial
  • Use inferencing as a way to suggest triples that would make the query simpler
  • Show the new query, and that the same answers are produced
  • Avoiding sparql/xquery inferencing
  • Is there a navigation process possible via existing widgets that would allow you to discover all related subjects and information you need, and produce a report off of the back of it?
  • Embed BI type summaries too? (E.g. facet graphs from document query results)
  • Sparql 1.1 in MarkLogic 7
  • Show what parts of the spec are there, and which ones we're missing, but which ones can be provided with a little work
  • Inferencing - we've shown, but should mention here
  • aggregates (count, sum, avg, etc.)
  • Sparql update

Unscheduled

  • Core: More support for server management (namespaces, indexes, service extensions, update REST instance configuration, XSLT transform management)
  • Widgets: Administration widgets (mainly aimed at pre-sales developers, not sysadmins)
  • Core: CSV and TSV utilities
  • Core: support arbitrary XPath for searches (if possible)
  • Core: allow upload and use of XSLT for result transform

Not applicable / dropped / postponed targets

  • N/A sensible to/from json settings (Default on &format=json on server)
  • N/A documented on WIKI - Generate JavaScript API docs and publish somewhere browseable on the web
  • N/A Support nested cts:search (if possible)
  • N/A (We use JSON EXCLUSIVELY for configuration, XML only appears in errors and search result content) - Samples for XML use, equivalent to some of the existing JSON samples (not exhaustive)
  • Widgets: Abstract enough to plug in Version 5 REST Wrapper results, as required (Corona?)
  • NodeJS: Provide REST API proxy within NodeJS wrapper
  • Browser: Support REST API proxy within general javascript and nodejs javascript code
  • NA generic xml document tree fallback
  • NA Widgets: Lazy loading on search results (E.g. when requesting page 2 result 3 (result 13), make http request for re-search transparent) - Also for graphing when all search results need loading

Past Releases

Targets for V 0.2 (Dec 2012) in descending order of importance

  • DONE abstract authentication (digest and basic)
  • DONE support creation of database and rest api on the fly
  • DONE provide document save, get, update
  • DONE access all documents within a directory (mljs.list)
  • DONE access all documents within a collection (mljs.collect)
  • DONE add ACID transaction support across calls to the db
  • DONE provide access to document properties fragment (all metadata)
  • DONE basic searching (aka search:search), with facets
  • DONE heavily document ALL quick start examples, with task orientated alternatives, and link back to REST docs on website
  • DONE handle errors elegantly
  • DONE support simple json structured query
  • DONE generic do function to invoke any REST API functionality I've not created a comprehensive helper method for
  • DONE Complete testing of all core functionality

For more information See the wiki.

Targets for 0.4 (Feb 2013)

  • DONE Utility function to save many docs in one go. (TODO: make transaction and fast aware)
  • TEST Create a saved search with a name (default grammar, collection, geospatial proximity) (NB uses REST API Extension from myself)
  • TEST subscribe/unsubscribe to saved searches (NB uses REST API Extension from myself)
  • TEST no auth (default user)

Targets for 0.6 (Apr 2013) - for MarkLogic World 2013

  • DONE Browser: Execute mljs from within general JavaScript, not just NodeJS (requires many wrappers, refactoring, extra tests)
  • DONE Core: search options persistence support
  • DONE Core: structured search support
  • DONE Widgets: basic demonstration samples for graphs based on MarkLogic data (Highcharts - shipped with MarkLogic)
  • DONE browser distribution building script
  • DONE Widgets: graphs to support simple aggregations of query results (mean, sum, min, max, count)
  • DONE Widgets: example for joining search bar to graphs, using different aggregations for the same results
  • DONE Widgets: Co-occurence widget
  • DONE Core: Added values() function for lexicon and co-occurence (GET /v1/values/[name])
  • DONE Widgets: Google Kratu Analytics widget linked to MarkLogic search results
  • DONE Core: include XML -> JSON transform in JavaScript (workaround for XML snippets / raw returned in search results even with format=json enabled)
  • DONE Widgets: Pretty print utility for errors in JSON and XML (Probably requires XML -> JSON utility. Errors are always XML, unless configured on server otherwise.)
  • DONE Widgets: Update search samples to include a custom renderer for one document type

Targets for 0.8 (Jul 2013)

  • DONE Added Apache 2.0 license text file and statement to all code files (not sample files though)
  • DONE Core+Widgets: Create search context object, and make search widgets responsible for registering themselves with it. Refactor search execution code in to here. (Required for Sort widget)
  • DONE Widgets: Support multiple values for save facet in facet widget and searchbar/context
  • DONE Widgets: Charts widget to support aggregations via search options (facet values)
  • DONE Widgets: Script to install latest MLJS in to an existing Roxy project
  • DONE Samples: ml-config.xml to include range indexes required for sample application
  • DONE Samples: Upgrade embedded Roxy to latest dev branch with ML7 EA3 nightly build deploy fix
  • DONE Tutorial: Deploying the mldbwebtest roxy project (text)
  • DONE NodeJS: NodeJS regression testing
  • DONE Widgets: Search widgets, and overarching search page widget UI layer over mljs in browser, including custom result rendering plugins (search bar, facets, results, paging, sorting)
  • DONE Widgets: Sorting widget actions
  • DONE Widgets: More... all... links in facets -> BUG always updating last facet, not the correct facet
  • DONE BUG: Pagination no longer refreshes search results shown (was probably related to JSON being saved as binary incorrectly)
  • DONE BUG: Hitting enter in search bar executes search
  • DONE BUG: one page too many in results if results are multiple of page size (i.e. 30 gives you 4 pages, not 3)
  • DONE BUG: multi word facet selection causes search to be limited to first word only (no quotes) -> query parser, not facet issue
  • DONE Core: support XML returned as well as JSON
  • DONE Search result content (XML as text within JSON results, not as XML results)
  • DONE Widgets: Update search samples to include one set of mixed XML and JSON documents
  • DONE Widgets: File upload (single)
  • DONE Core: Search Options Builder
  • DONE Browser: XMLHttpRequest (xhr) binding
  • DONE Browser: XmlHttpRequest2 (xhr2) binding
  • DONE Tests: Updated mldbwebtest project to be more obvious, and include content initialisation on main page
  • DONE Widgets: Search results rendering default processor
  • DONE Browser: Move SVG rendering code in to default search results renderer from animals sample
  • DONE Widgets: Get snippeting mode working
  • DONE Test: Snippet search mode test page
  • DONE TEST xhtml documents
  • DONE generic xml document (//title, //summary, //description, //synopsis, or first 2 elements, or xml snippet)
  • DONE generic JSON document tree fallback
  • DONE Tests: Test samples for xml content
  • DONE hurt-bat svg in animals search test
  • DONE Docs: Architecture of widgets and MLJS for developers -> Issue with MLJS gh-pages docs rebuild process, not linked properly
  • DONE Integrate error widget on each demo page
  • DONE Add button to allow force reload on main page
  • DONE Index detection and reporting on main page
  • DONE Tests: Refactor to use alternate testing framework (Mocha)
  • DONE Core: Return and add binary documents
  • DONE Core: Return and add plain text documents
  • DONE EA 1 support
  • DONE TEST Add/delete/modify/replace named graphs
  • DONE IN PROGRESS Interactive SPARQL query builder and result handler (including paging, linking to document search, restrict by collection, other document search criteria)
  • DONE Get collections starting with a URI (values query helper method)
  • DONE Rename all to MLJS from mldb

Targets for 0.9 (Sep 2013)

  • DONE Interactive RDBMS migration utility
  • DONE REST extension: POST stuff2triples with JSON config specifying what to ingest
  • NA make long transaction aware
  • DONE add to MLJS Core
  • NA MLSAM presence detection / error message
  • DONE Widgets: Interactive migrator
  • DONE Pg 1: Enter DB connection configuration parameters - NA Use cookies to hold DB connection parameters in user's browser so its quick to perform both imports
  • DONE Pg 2: Select which tables & relationships to ingest - DONE (with totals of records to import, column data type conversion to use) - DONE Also allow specifying a named graph to add all this new data to
  • DONE Pg 3: Show 'importing 1 to 100 of y' message for each table, synchronously, all within the same transaction - DONE Widgets: Progress bar widget (done via ticks in a table) - NA Generate and save ontology information in cookies too - NA Create semantic context decorator widget to fill in semantic context based on ontology information available within user cookies (So new objects show up in Sparqlbar immediately)
  • DONE Core: Search Context
  • DONE Structured Search (required for Sparql semantic context's linking to fact provenance documents - derived_from)
  • DONE Structured Search Contribution mode - allow multiple widgets to contribute a query to the content context that is AND'ed with other contributed queries and submitted
  • DONE IN PROGRESS Widgets: Sparql: Various SYTYCD alterations
  • DONE BUG REGEX suggestion match for Ada* should have been Ada.* in order to produce correct results (trailing letter was being interpreted as a ., producing erroneous results)
  • DONE Check returned triples for type (URI or value) and display appropriately
  • DONE Drive property list equals values dynamically using sparql (suggestions feature via FILTER regex)
  • DONE Support DISTINCT (always eliminates duplicates)
  • NA Use OFFSET and LIMIT to do paging for subject results sparql query
  • NA Used inferencing step to generate JointCustomer triples - Figure out how to do sameas - either as separate inferencing stage, or dynamically within the query (Introduces the concept of Sparql Query Options)
  • NA Widget: Inferencing with named graph for output, sparqlbar for production of constraints?
  • NA Add named graph restriction support for facts (So we can just query triples generated from DB 1 or DB 2)
  • NA MEDIUM Add paging to subject results widget itself
  • NA LOW Add preview quick view to subject results widget
  • DONE Move summariseInto out of tripleconfig and in to widget library
  • NA Support ORDER BY
  • NA Use ?s ?p ?o ?g rather than longer ?subject ?predicate ?object ?graph, everywhere
  • DONE Widgets: CSS: Add bootstrap.js support for Jochen
  • DONE searchbar - uses btn and btn-primary classes
  • DONE sparqlbar - uses well, btn and btn-primary classes
  • DONE Widgets: Kratu
  • DONE updateFacts generic method added to MLJS core's semantic context - Support viewing arbitrary sparql results as Kratu table (needs reworking of bindings result object to a flatter classic row structure)
  • DONE Minor fixes
  • DONE Remove debug logging for every value in charts
  • DONE Samples: Sample triples, linked content, working sparql query builder
  • DONE Widgets: Semantic rearchitecting
  • DONE addTriplesListener to replace addResultsListener
  • DONE Core: rework searchcontext to handle triples listener
  • NA we specify this through an Accept field - Core: mljs core to check /v1/sparql result for triple format type and place in result.tripleFormat (owl, n3, etc.)

Targets for 12 Sep 2013

  • DONE Core: Document context for document viewing (not editing) type widgets
  • DONE Widgets: XHTML Document view
  • DONE Get Bootstrap JS working in mldbtest
  • DONE Fix widgets.css and one-column.less for working with bootstrap
  • DONE (Compatible with V6, JSON only) Widgets: Document properties viewing, and editing (via uriHandler, using patch in V7?)
  • DONE JSON only. Check that mljs.properties in default JSON format mode actually works as expected. May be better to always default this to XML
  • NA? May also be an idea to allow XPath selection of the subset to return, and allow conversion of that to JSON, if possible
  • DONE Draw nice boxes around options
  • DONE show modal popup for editor
  • DONE test save function
  • DONE add property removal function (remove all values, property is removed)
  • DONE remove property option from allowed properties UI widget if one has been added
  • DONE Highlight/underline metadata you can edit -> Obvious edit button? Always underlined? (nah that'd look shabby)
  • TEST support very long, line wrapping URIs in docheadviewer title
  • IN PROGRESS Firefox transparency issue - modal dialog is still transparent!
    • Trying to use modal.css to get around issue
  • DEFERRED Widgets: Folksonomic keyword editing with auto suggest support
  • DEFERRED Core: Lexicon suggestions on V6
  • DEFERRED documentcontext (for properties)
  • DEFERRED searchcontext (for any constraint/lexicon)
  • DONE Core: setProperties using V6 JSON
  • DEFERRED Core: replaceProperty using V7 PATCH /v1/documents
  • DEFERRED Core: All properties (+XML, +all JSON types) support
  • DONE Search Structured Query: Support for constraints in properties fragment scope
  • DONE Search by keywords (use keyword property facet)
  • DEFERRED Nice to have: Widgets: Find similar documents link support

Targets for 22 Sep

  • DONE Selection of search results across pages
  • DONE Search context to support multiple selection
  • NA Button widget to respond to selected slides (becomes active upon selection)
  • DEFERRED change colour to indicate result is selected
  • DEFERRED highlight previously selected results in search results
  • DEFERRED Date facet with buckets and selection of range
  • DEFERRED Facet bucketed by date?
  • DONE Profile info widget
  • DEFERRED Save a search, view in profile page
  • DEFERRED Tag cloud based on facet and size (D3.js + SVG)

Targets for 27 Sep

  • DONE Semantic/Data Explorer - HighCharts
  • DONE basic view working
  • DONE facets view working
  • DONE Link to originating SPARQL query for subject list
  • DONE Lazy loading of facet values when added to output

Targets for 1.0 (1 Nov 2013) (Published early due to serious changes to V7 affecting functionality)

  • DONE Rework MLDBWEBTEST sample app so Dave Cassel can easily grab example controller for Roxy auto deployment
  • DONE rename page js files to be page-controller-action.js
  • DONE Move JS, CSS files to correct positions
  • DONE Update build scripts to take new locations in to account, including mljsme
  • DONE Rework layout files for Roxy for MLJS core including all widgets
  • DONE JS Health Check
  • DONE Perform Firebug comparison to check on speed of widgets
  • searchpage (170ms, 14.5ms in widgets)
  • searchpage+snippets (163ms, 10.7ms in widgets)
  • charts (1273ms,<20ms in MLJS)
  • charts+search (1003ms,<20ms in MLJS)
  • cooccurence (173ms,0.873ms in widget)
  • error (153ms, 0.597ms in widget)
  • kratu (153ms, 0.237ms in widget)
  • viewdoc (162ms, 1.2ms in widget)
  • sparql - not tested - was on v6
  • explorer - note tested - was on v6
  • workplace (613ms, 12.2ms in widgets - 2 actions, load plus facet selection)
  • DONE V7 update fixes
  • DONE BUG: search results JSON content being returned as String not object in recent V7 builds (workaround for both approaches)
  • DONE tripleconfig ontology builder
  • DONE Added rdftype() function to better manage adding new Entity classes to an ontology (Works like the options builder object, with method chaining)
  • DONEAdded include() function to take this JSON configuration and apply it to the tripleconfig object
  • DONE JavaScript public API consistency changes
  • DONE UpperCamelCase for constructors/objects - only when publicly visible
  • DONE Core API
  • DONE Widget API
  • DONE lowerCamelCase for functions (searchcontext and mljs have some alllowercase methods)
  • Apply Configuration object pattern consistently with MLJS Core (E.g. search options), and add helper methods where this is complex for common use cases (location 1734)
  • DONE alias options.sortOrderScore() to options.relevance()
  • support snippet size too (extended snippets) in search options builder
  • DONE mljsme.sh improvements
  • DONE Copy dependant files - 960 CSS folder, bootstrap-roxy.css, one-layout.less, mljs-one-layout.html.xqy, highcharts.js
  • DONE Check for wget and/or curl presence, and include helpful doesn't exist message
  • DEFERRED for Workplace in 1.2: add option to also deploy -workplace mljs/main (Workplace) and -test mldbtest/* for retrofitting Roxy apps
  • DONE Alter build.sh to include roxy/ folder with controller, layout, css
  • DONE Core: TripleConfig improvements
  • DONE rewrite how ontology information is provided, configured, accessed and managed - far too complex and buggy for the average developer at the moment
  • DEFERRED TripleConfig internal array and function rewrite to remove duplication of predicate information
  • DEFERRED Support loading of OWL / other ontologies to populate triple config object
  • DEFERRED Support prefix configuration in tripleconfig
  • DONE ontology builder object - use methods on tripleconfig
  • TEST Widget: Error
  • DONE Ensure it works with result objects (not result.error objects) and javascript errors
  • TEST Ensure this is invoked correctly, and all contexts and test pages work with it
  • App Builder gap closing
  • DONE via sprops_opt.transform and .transformParameters - Support for XSLT transformations on search results
  • DONE via options_opt.transform and .transformParameters - Support for XSLT transformations on fetch documents in new window
  • DONE via sprops_opt.transform and .transformParameters - Support for XQuery transformations on search results
  • DONE via options_opt.transform and .transformParameters - Support for XQuery transformations on fetch documents in new window
  • TEST Support for bucketing in facets (all types, including dynamic)
  • DONE Just pass in as raw facet_options_opt string array: Facet ordering (other than frequency)

Targets for 19 Nov 2013

  • DONE Widgets: OpenLayers phase 1
  • DONE Basic map display
  • DONE Google Street maps
  • DONE OSM
  • DONE offline caching (load cached tiles by default) - load tiles, keep browser open, then go and demo on customer site using same zoom level!
  • DONE updateResults support (show pins for locations)
  • DONE Core: Geospatial search options (via constraints)
  • TEST geo-element-pair-constraint
  • TEST geo-element-attr-constraint
  • DEFERRED geo-element-constraint For location: "lat,lon" type elements
  • DEFERRED geo-path-constraint (V7)
  • DONE Test: Geospatial search options in mljstest/main data loading page
  • DONE Test: Add data for tourist attractions in london
  • DONE Test: Alter roxy config to include index for lon, lat
  • DONE Test: EPSG900913 to/from WGS84(EPSG:4326) automatically - see http://en.wikipedia.org/wiki/World_Geodetic_System
  • NA Simply don't facet the constraint: Widgets: Geospatial facets shown in maps, and not in facets widget
  • DONE Core: V7 range relevance by distance from location in structured search
  • DONE For distance of tourist attraction in sorting
  • DONE Widgets: Structured query selection widget
  • DONE Select Relevancy method. E.g. 'Distance from point' or 'Star rating'
  • DONE Requires providing a function to determine values for query at query time, rather than hard coded (E.g. point for 'distance from point') (called query.dynamic())
  • DONE Core: Structured Query geo alterations
  • DONE geo-elem-pair-query (fully dynamic, no constraint options required) + dynamic query support, for points and circles (radius)
  • DONE Widgets: OpenLayers phase 2
  • DONE BUG: attribution appears 1/4 way up the map (CSS issue)
  • DONE Added ArcGIS Online and Bing (road, hybrid, aerial) map layer convenience functions
  • DONE Area selection and query
  • DONE Core: Add updateGeoSelection detection to searchcontext
  • DONE Widgets: OpenLayers - Configure openlayers widget with name of constraint to provide query for on selection
  • DONE Core: Add polygon, bounding box geo queries to query builder
  • DONE Widgets: OpenLayers - support bounding box
  • DONE Widgets: OpenLayers - support polygon
  • DONE Widgets: OpenLayers - clear selection button
  • DONE BUG: Contribute query does not include one selected from drop down
  • DONE BUG: When clearing query, not contributed query is used at all (just two nulls)
  • DONE Widgets: OpenLayers - Configure OpenLayers/test page to use structured Contrib mode for queries
  • NA using Highlight instead due to popups being poor in OpenLayers: Show number with pin
  • NA using Highlight instead due to popups being poor in OpenLayers: Details popup for pins
  • DONE 'highlight' support for search result aka click on pin (not mouse over)
  • DONE 'select' support for search result aka 'details' link in description popup
  • DONE Polygon, Circle, Rectangle drawing and generation of query contribution
  • DONE Heatmap support (tourist attractions per km type result)
  • DONE update heatmap constraint in options document
  • DONE return heatmap info in facets
  • DONE Include heatmap.js and heatmap-openlayers.js files, including updating build scripts
  • DONE Process heatmap result: in facet location.boxes (location was the constraint name). Each box has count, n, s, e, w. zero count boxes not returned
  • NA Dependant upon base layer, cannot remove if present: Allow hiding of heatmap layer
  • DONE Allow configuring of heatmaps - use setHeatmapGranularity()
  • DONE Update heatmap options and perform search again on map zoom (zoom NA - calculated by heatmaps js on the fly)
  • NA No need - Change to using shadow query just for heatmap?
  • NA Why? Works fine - Change to use geoConstraintName(cn) as the heatmap constraint name (not just constraint for additional openlayers positional query)
  • DONE Change to automatically calculate boxes in lon/lat planes. E.g. LOW=64 boxes, if 800x800 map, this would be 4 wide and 4 tall, MED=128, HIGH=256
  • DEFERRED Support RAW mode - extract lon/lat metadata ONLY, and pass all results directly to heatmap.js code (rather than calculate within MarkLogic)
  • NA can add this external to map if needed (using addGeolocate function) - Locate me button support - http://openlayers.org/dev/examples/geolocation.html
  • TEST Locate me automatic (browser) with no button
  • DONE Fix dynamic width/height detection (el.offsetHeight)
  • DEFERRED Listen for structured query parameter and draw search radius (E.g. 20 miles)

Targets for 1.1 Interim release (Nov 2013)

  • DONE Widgets: Tag Cloud for a facet
  • DONE Allow sorting by name ascending (currently bubblesort only works descending)
  • DONE App builder gap closing
  • DONE Ensure 'Sort by relevancy' shows in sort drop down, when configured in the search (achieved via 'states' in AppBuilder)
  • DONE Check for sort order annotation[0] to use as title rather than automatically determining it from element name/order
  • NA Achieved via Workplace not alterations to searchpage - Embedding charts within search page widget like AppBuilder
  • NA Achieved via Workplace not alterations to searchpage - Support for multiple layouts in search page
  • DONE Phrase searches - surround with "some quotes"
  • TEST Support Fields for range constraints
  • DONE BUG: Widgets: HighCharts - handle 'undefined' in label on pie charts (only happens on chartsearch page, not on workplace page)
  • DONE dynamic and static: Date time bucketing
  • DONE Widgets: Navigable charts / co-occurence - clicking sets facet value
  • DONE Co-occurence (Structured query mode only, not via searchcontext) - DONE BUG: Default constraint options handling (set up search options method?) is causing tuples to return nothing every time
  • DONE Core: Add annotation support in constraints and facets functions
  • DONE Use annotations in widgets for facet name
  • DONE Sort widget
  • DONE searchfacets widget
  • DONE Docs: date time dynamic bucketing example in search page sample

Targets for 1.2 (Dec 2013)

  • DONE Widgets: searchfacets - allow developer to specify facet value to human name. E.g. VALUE/hello -> Hello (still using "FACET/hello" as the actual value)
  • DONE Core: Change options builder to store text translations for facets
  • DONE Core: Change searchcontext so that setOptions takes an options builder (as well as legacy options JSON), so as to maintain a reference to these transactions
  • DONE Widgets: Address to Lon Lat - Perform address lookup for top matching point, feed as constraint in to structured query
  • DONE Google address lookup
  • NOT A BUG wasn't using same searchcontext - need to change setContext to setSearchContext globally - contributing multiple terms only processes last term in search
  • DONE BUG heatmap doesn't clear when zero results returned from next search
  • DONE Accept radius as a property
  • DONE Core: Add GeoContext to support concept of Area(s) of interest (E.g. total area that should be visible on a map, or within an area for analysis)
  • DONE Widgets: Update addressbar to support geocontext
  • DONE Docs: Update address lookup search page to use this
  • DONE Widgets: Update OpenLayers to zoom to the location mentioned
  • DONE Docs: Update openlayers map page to use this
  • DONE BUG Widgets: OpenLayers heatmaps appear offset and are not refreshed when map's center position is changed
  • DEFERRED Widgets: Update OpenLayers to zoom to the bounds mentioned (not just the center point at the current zoom level)
  • DEFERRED Core: Calculate max bounds from areas specified and pass that in the updateLocale event
  • DONE Docs: XML example for custom renderer support, with namespaces in XPath
  • NA Not happening now: Figure out why HTTP 500s result in throwing an exception rather than MLJS error handling (e.g. extension execution errors)
  • DONE Ensure version extension absence errors gracefully
  • DONE Rework all examples to use searchcontext rather than direct invocation of mljs methods
  • DONE need to use values() not search() - BUG cooccurence widget now not returning results in combined V7 query mode
  • DONE Try N-way cooccurence (See co-occurence widget test page)
  • TEST Core: Options Builder
  • TEST Added all options support for geoElement, geoElementPair, geoAttributePair and geoPath Constraints
  • BUG geo-elem-pair constraint seems to ignore fragment-scope=properties option (reported to ENG) - 25342 in 6.0-5
  • TEST Add custom constraint support
  • TEST Add thesaurus support using custom constraint
  • TEST element query constraint
  • TEST Add extract metadata support (For use by custom renderers)
  • TEST searchable expression
  • TEST Add custom grammar support
  • TEST Top level options properties: search-option,
  • DEFERRED (default)suggestion-source
  • BUG App builder gap closing
  • DONE Widgets: HighCharts: Apply chaining pattern to highchart options to make them simpler
  • BUG Core: Rework searchcontext
  • DONE Allow contribution of facets/sort etc to alter both text (grammar) queries and structured query (independently, via searchcontext)
  • DONE Add "values" as a searchcontext endpoint option (either /v1/search (default) or /v1/values/{tuplesname})
  • DONE Added custom REST endpoint support to Search Context, and a tutorial on this to GitHub. See searchcontext.customEndpoint()
  • DONE Add V7 combined options + values support to mljs.values()
  • DONE Ensure searchcontext supports updateFacets
  • NA Done when query refreshes anyway: Add addFacetSelection listener etc to highcharts widget
  • IN PROGRESS Support compound search mode in V7 (options + search) (search, searchCollection, structuredSearch, saveSearchOptionsCheck, searchcontext alterations) - DONE version rest extension - BUG: V7 ignores URL search parameters if submitting a structuredQuery and options (previously if supplying just query and referring to options, it'd work) - 25306 in 7.0-2
    • Fix chart search page
  • DONE Docs: Rework cooccurence example to use search context, and contributeStructureQuery
  • NA Not possible. Close approximation without grammar with term queries though: Check if structured query supports dynamic text query constraints with grammar (i.e. one term with grammar parsing)
  • DEFERRED Do shadow query automatically in searchcontext rather than per-widget?
  • DEFERRED Come up with a strategy for parent-child relationships in search widgets (one way facet selection - E.g. genre THEN album)
  • NA alter the options to fetch data. Use updateResults to retrieve them for display: Allow widgets to request information in results, rather than alter search options (E.g. request a heatmap, or particular metadata properties) - context updates options on the fly
  • DONE Core: SearchContext to support multiple searches when same context used across multiple tuple names
  • BUG Widgets: Navigable charts / co-occurence - clicking sets facet value
  • BUG co-occurence - DONE Requires values() support in searchcontext - BUG POST /v1/tuples also ignoring query string parameters - 25306 in 7.0-2
  • DONE High charts column
  • DONE High charts line
  • DONE Pie segment
  • DONE translated to category from x position: BUG passing 2:"8" - 2 is the x position, not the x value - should be 'pet' on the pie on chartsearch page
  • DONE BUG: OpenLayers polygon/circle selection no longer removing markers when executing more restrictive search
  • DONE BUG: Search sort widget showing [Object: object] instead of name in drop down
  • DONE BUG: Widgets: Search results. Plain text parsing error.
  • TEST mljs.combined and mljs.doCombinedQuery. Auto on contributeStructuredQuery already (if term is text).
  • TEST Core: searchcontext - support qtext for combined query when searchbar is contributing information to a structured query (rather than a term query as now)
  • TEST How to handle this on V6 only systems? - Detect and fall back to term query
  • DONE Widgets: Tag cloud - add hover over text to include frequency of result
  • IN PROGRESS Widgets: Workplace - Removes any code anyone needs to write to use MLJS widgets
  • IN PROGRESS Main edit page framework
  • IN PROGRESS Layout rendering in edit mode - DONE Render layout - IN PROGRESS Render widget configs within layout
    • DONE Replace encodeuri with encodehtmltag
    • DONE http 500 error handling added to xhr2 library: Figure out json parse bug on loading main page (taking up too much debug time)
    • Remove Data Series (Array of settings) Button: Data Series in table drawing
    • IN PROGRESS Get button images and css sorted for multiples
    • DONE bad widget config: BUG chart type options - only showing Line
    • DONE shrink drop zone / make look better (sort out images)
    • DONE add Data series instance number
  • IN PROGRESS saving previous tab settings on switching tabs - DONE page settings - IN PROGRESS actions
  • DONE Page (configuration) settings - DONE Get workplace rest extension and wrapper - DONE Save workplace rest extension and wrapper - DONE Workplace config JSON format - including 'shared' flag, title, description - DONE visuals (drop downs) - DEFERRED no way to search for shared yet anyway: shared checkbox
  • IN PROGRESS Widgets settings - DONE Widget toolbox - DONE Laying out widget positions within page Layout - IN PROGRESS Add new widgets and place correctly when dropped
    • DONE Drag/drop class on bottom drop zone places new widget correctly
    • BUG drag/drop new or existing widget on other drop zones makes them initially appear at top, then split from their spacer
    • Rework layout code so there's only a single method to add a wrapping element - createWrapper() ??? what about layout.wrap(widget)? layout.register(widget)? - DONE Add 'Insert here' before each widget too - TEST Make existing widget titles drag/droppable too - TEST Save changes to layout configuration on drop - TEST Add html moving code, preserving event handlers - TEST Apply equivalent change to workplace context when saving positions (switching away from widgets tab or saving) - TEST Break out thinthick logic in to generic mixin for layouts
  • IN PROGRESS Widget configuration settings (in place) - IN PROGRESS visuals - Contexts - add new (auto named)
    • Thinking of doing this on the left hand side still
  • Action settings - IN PROGRESS on page load - IN PROGRESS on page unload - DEFERRED searchcontext on selection
  • Widgets: Add Workplace Configuration support - co-occurence - OpenLayers - Address lookup - TagCloud - document (result) selection, search selection - searchbar - sparqlbar / sparqlresults / entityfacts - searchmetrics - graphexplorer - documentproperties / docheadproperties / docviewer - docbuilder (upload) - kratu - rdb2rdf (no configuration params needed) - layouts (embedding) - DONE thinthink / TEST thickthin / TEST single column / double column / triple column / 2 rows / 3 rows
  • DONE Programmatic access
  • NA Doesn't make sense - all handled internally by workplacecontext: getWidget by name (so you can programmatically add settings after load)
  • DONE getWorkplaceContext (could be done via widget optionally too)
  • NA done in real time (run til complete): addLoadedListener (so you can programmatically add settings after load)
  • DONE addSupportedWidgets() - so you can add widgets allowed to workplaceadmin after load
  • DONE addSupportedLayouts() - so you can add layouts allowed to workplaceadmin after load
  • DEFERRED 'app level' settings - configured search options
  • DONE Widgets: Add Drag and Drop support to widgets.js
  • DONE DND pure Test page in mldbwebtest roxy project
  • DONE ensure basic drag and drop works
  • DEFERRED Only needed when dnd used by third party developers on complex apps: Add rules around which draggables can be dropped on which droppables (dropzones)
  • Render Workplace
  • (default) Detect URL of page. Search in ML for workplace json page that matches this URL (not including query strings)
  • User default workplace. Find user's own 'personal-workplace' for currently logged in user (akin to a single page mode app)
  • DEFERRED Embeddable. Assume embedded in external app. Use workplace-share-id as key to find, on remote server via W3C CORS, the 'page' (embed) to draw
  • Fallback. If no Workplace detected, create a generic search page with a Workplace edit button (depending on app wide settings - 'save/edit allowed') - Allow fallback to be set programmatically or via app wide settings
  • DONE Create WorkplaceContext object in workplace js file
  • DONE updatePage event - occurs when a page is loaded. E.g. from current page URL or for current logged in user or as fallback
  • NA using updatePage instead - updateSummary
  • DEFERRED until we support app wide settings - updateApp event
  • DEFERRED until we support app wide settings - loadApp
  • DONE loadPage
  • DONE Latest REST API returns JSON as object not text: BUG Kratu no longer rendering content since migration to searchcontext
  • DONE Docs: Document all core concepts currently part of MLJS
  • DONE Docs: searchcontext object
  • DONE Docs: Supported rest extensions tutorial doc
  • DONE Core API jsdoc
  • DONE Widget API jsdoc
  • DONE Core: Search Options builder
  • DEFERRED hide sem:triple elements from snippet highlighting
  • DEFERRED Don't return sem:triples documents at all by default (Top level only)
  • DONE actually write the pathConstraint function!
  • DONE BUG WAS AN ISSUE IN MLJS CONSTRAINT SETTINGS: Investigate why REST insists upon some elements having indexes in the JSON namespace as well as the one required (E.g. xhtml and h1)
  • DONE Test: Create MLJS Rest only (not Roxy hybrid) deployment project
  • DONE Docs: Publish mljsrest and mldbwebtest to internal demo server
  • DONE Bug scanning/regression testing check
  • DONE BUG Widgets: Default Search Renderer does not check for format="xml"!!! Assumes JSON even when format is specified
  • DONE Widgets: searchresults add default renderer for documents containing triples (sem-triples parser)

Targets for 1.4 (Mar 2014) - Theme: Documentation and Tidying up

  • DONE Updated tutorials
  • DONE Use more effective jsdoc theme (Cerulean from docstrap)
  • DONE added one day introduction course material (in tutorials)
  • DONE Widgets: Search Results: add support for 'advanced renderers' - render sections of search results summary. E.g. title, snippet/summary, related docs, thumbnail etc.
  • DONE all (as now)
  • DONE overarching default search results layout abstraction
  • DONE default search layout CSS
  • DONE default result summary (short - snippeting or non snippeting)
  • DONE BUG: existing results showing is screwed up (renders nothing) - FIX: was parsing JSON and XML incorrectly in mljs-xhr2.js
  • DONE metadata view section below search results by default
  • DONE Reusable string generating calls on manager. E.g. title, summary, metadata (generates correct div tags and style links)
  • DONE Test this against existing 'old style' renderers
  • DONE Test mixing custom advanced renderers with default section renderers
  • DONE Core: Search context
  • DONE Core: Add metadata extraction support to search options and search context - extract() method on context?
  • DONE Core: Add generic XML to XHTML support and CSS via XSLT transform in REST API
  • DONE BUG Widgets: address bar: generates blank point - google geocode API has changed param names for lat and lon
  • DONE BUG build script: Ensured mljsrest project is updated with every build