Ordered events #68
Merged
Ordered events #68
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Replace Deno-specific asserts with Node.js compatible versions - Implement flexible ErrorClass type for assertRejects - Update datasetIssues.test.ts and loadSchema.test.ts to use new assertions - Add explanatory comments to clarify design decisions
- Remove Deno-specific dependencies: * Replace cliffy.ts with npm:commander * Remove fmt.ts (no direct replacement needed) * Replace prettyBytes.ts with custom function - Update affected files: * options.ts: Implement argument parsing using Commander * options.test.ts: Adjust tests to work with new Commander-based parsing * output.ts: Modify to use new dependencies and remove Deno-specific code Key changes: - Improved CLI argument handling with Commander - Removed Deno-specific formatting utilities - Updated byte formatting to use a custom function - Adjusted related tests to ensure compatibility with new implementations
- Replace Deno logger with custom Winston-based logger - Update logger implementation in logger.ts: - Add Winston configuration with custom formatting - Implement proxy handler for additional functionality - Maintain compatibility with existing log level exports - Update logger tests in logger.test.ts: - Add MemoryTransport for capturing logs in tests - Expand test coverage for all logger functions - Include tests for stack trace parsing and log levels - Prepare codebase for dnt build and npm publication - Improve type checking throughout logger implementation
- Replace Deno logger with custom Winston-based logger - Update logger implementation in logger.ts: - Add Winston configuration with custom formatting - Implement proxy handler for additional functionality - Maintain compatibility with existing log level exports - Update logger tests in logger.test.ts: - Add MemoryTransport for capturing logs in tests - Expand test coverage for all logger functions - Include tests for stack trace parsing and log levels - Prepare codebase for dnt build and npm publication - Improve type checking throughout logger implementation
- Replace custom path.ts dependency with node:path module - Update import statements in multiple files: - deno.test.ts - deno.ts - context.test.ts - walk.test.ts - options.test.ts - filenameIdentify.test.ts - filenameValidate.ts - psychds.test.ts - main.ts
- Remove import of readAll and readerFromStreamReader from deps/stream.ts - Update psychDSFileDeno.text() method to use native stream handling - Add try/finally block to ensure decoder is always flushed - Improve error handling for UTF-16 BOM detection - Modify 'can be read as ReadableStream' test in deno.test.ts - Replace Buffer.concat with Uint8Array for cross-platform compatibility - Implement manual chunk concatenation - Add explanatory comments to clarify changes and their rationale - Enhance resource management to prevent potential memory leaks
Changes to loadSchema.ts: - Implement environment-agnostic directory path resolution - Enhance error handling and logging - Add stricter version format validation - Implement local fallback schemas (defaultSchema.json, defaultSchemaOrg.json) - Improve TypeScript typing and code organization - Refactor to use node:fs and node:path for better Node.js compatibility - Implement dynamic import of node-fetch for cross-platform fetch support Changes to loadSchema.test.ts: - Add comprehensive tests for version format validation - Implement tests for both valid and invalid version strings - Enhance existing tests for schema loading functionality New files: - Add defaultSchema.json and defaultSchemaOrg.json as fallback schemas
- Modified csv.ts to use 'csv-parse/sync' from npm instead of Deno's std/csv - Updated CSV parsing options to handle empty lines and inconsistent column counts - Improved error reporting for header row mismatches - Fixed related test issues in deno.test.ts and applyRules.test.ts - Removed empty lines between rows in test_data/valid_datasets/face-body CSV files
- Modified csv.ts to use 'csv-parse/sync' from npm instead of Deno's std/csv - Updated CSV parsing options to handle empty lines and inconsistent column counts - Improved error reporting for header row mismatches - Fixed related test issues in deno.test.ts and applyRules.test.ts - Removed empty lines between rows in test_data/valid_datasets/face-body CSV files
- Created deno.json to centralize Deno configuration - Specified import_map.json location in deno.json - Defined 'test' task for running tests with appropriate flags - Eliminates need for explicit --import-map flag in commands and CI/CD - Ensures consistent import map usage across local and CI environments
- Changed npm:jsonld to jsonld in light of importmap changes
- Updated deno.json to include a lint configuration - Added exclude property to ignore custom_test_runner.js and other non-deno files related to dnt during linting
Modified files: - deno.test.ts - context.test.ts - walk.test.ts - filenameIdentify.test.ts - filenameValidate.test.ts - psychds-validator.ts - emptyFile.test.ts Changes made: - Refactored code to eliminate usage of top-level await - Wrapped async operations in functions or immediately invoked async functions - Ensured all asynchronous code is properly encapsulated
- Modified deno.json to add npm/ to the exclude field under test - Prevents Deno from attempting to run tests in the npm build output directory
- Modified main function to accept args parameter, defaulting to Deno.args - Added error handling for options parsing - Improved function signature to support both Deno and Node.js environments
- Modified loadSchema.ts to use global fetch instead of node-fetch * Ensures compatibility across Deno, ESM, and CJS environments - Updated loadSchema.test.ts to use mock responses instead of real fetches * Resolves memory leak issues in tests * Improves test reliability and speed
- Resolved 'require-await' error in mock fetch function - Addressed 'no-explicit-any' warning by using 'unknown' type - Fixed 'no-prototype-builtins' issue with Object.prototype.hasOwnProperty.call()
- Updated 'fileTree exists' test step to properly verify file tree structure
- Updated index.ts, psychds-validator.ts, and validate.ts (and removed main.ts) - Separated concerns between Deno and npm (ESM/CJS) environments - Enhanced validate() to work with both CLI arguments and import contexts - Tailored psychds-validator.ts specifically for Deno usage - Added comprehensive comments and docstrings for improved readability
- Removed separate importmap.json file - Added imports field directly in deno.json
- Removed separate importmap.json file - Added imports field directly in deno.json
- Update options.ts to use lowercase for debug level parsing - Modify options.test.ts to use lowercase debug levels in assertions
- Update validate.ts to include dataset path in args array when fileTreeOrPath is a string - Ensure the dataset directory argument is properly passed to parseOptions() - Resolve 'missing dataset_directory' error occurring in downstream processes
- Add build_npm.ts to transform Deno codebase into npm-compatible packages - Create shims.ts to handle Deno-specific APIs in Node.js environment - Enable creation of ESM and CommonJS versions for npm distribution - Support usage as both global CLI tools and importable JS packages
- Add sanitizeOps: false and sanitizeResources: false to Deno.test options in logger.test.ts
- Add release-please job for version management - Include build-test-publish job for testing and npm publication - Create package.json for version tracking - Update deno.yaml to incorporate new CI/CD workflow - Update build_npm.ts to manually exit process when error is thrown (for CI/CD)
- Remove fileText property from psychDSFileDeno class - Remove JSON-LD parsing and expansion from _readFileTree function - Remove context handling for dataset_description.json - Simplify file reading process in _readFileTree This change reverts the earlier modification that added fileText and moved JSON-LD parsing to the readFileTree stage. It streamlines the file reading process and removes browser-specific accommodations.
…rom readFileTree - Remove fileText property and JSON-LD parsing from psychDSFileDeno class - Simplify _readFileTree function by removing JSON-LD expansion and context handling - Move JSON-LD expansion to getExpandedSidecar method in context.ts - Update applyRules.ts to work with the new structure, removing fileText references - Adjust psychds.ts to handle JSON-LD processing changes and error handling This refactor simplifies the file reading process, moves JSON-LD handling to a more appropriate location, and improves overall code organization and maintainability.
added 9 commits
August 28, 2024 11:59
- Remove fileText property from psychDSFile interface - Add expanded property to psychDSFile for storing expanded JSON-LD - Update Context interface to include validColumns and suggestedColumns - Add metadataProvenance to Context for tracking metadata sources These changes align the type definitions with the recent refactoring of file reading and JSON-LD processing, improving type safety and clarity throughout the codebase.
- Refactor applyRules.test.ts to use a setupTest helper function - Update test cases to work with the new file reading and JSON-LD processing flow - Adjust context.test.ts to handle the new psychDSContext structure - Update assertions to check for expanded JSON-LD properties - Remove fileText references and update to use new expanded property These changes align the tests with the recent refactoring of file reading and JSON-LD processing, ensuring continued test coverage and validity.
- Introduce ValidationProgressTracker class for real-time validation progress - Implement checklist-style output for each validation step and substep - Add event-driven updates for individual validation checks - Display success/failure status and issues for each step - Replace single completion response with continuous progress feedback This enhancement provides users with a more detailed and interactive view of the validation process, allowing them to see the progress and results of individual checks as they occur.
- Implement emitCheck function to standardize event emission for validation checks - Add emit calls for various validation stages: - start, build-tree, find-metadata, find-data-dir - metadata checks: utf8, json, fields, jsonld, type, schemaorg - CSV checks: keywords, parse, header, nomismatch, rowid, variableMeasured
- Implement useEvents flag in ValidatorOptions - Modify index.ts, psychds-validator.ts, and validate.ts to support event-based validation - Integrate ValidationProgressTracker for real-time progress updates - Add conditional logic to use EventEmitter when useEvents is true - Ensure backwards compatibility with non-event-driven validation This change allows users to opt-in to event-driven validation, providing real-time progress updates and improved visibility into the validation process. The existing synchronous validation remains the default behavior.
…amespace - Simplify JSON-LD manipulation in context.ts - Update schema.org namespace from https to http in both files - Improve type safety for JSON-LD related operations - Enhance error handling for JSON-LD expansion - Adjust schema.org property checks to use http namespace
…splay - Introduce useEvents flag in ValidatorOptions type - Add --useEvents command-line option to the CLI parser - Update parseOptions function to include useEvents in returned options object
- Modify options.test.ts to include useEvents flag in expected output - Update context.test.ts to use HTTP instead of HTTPS for schema.org namespace - Adjust assertions in context.test.ts to reflect new JSON-LD expansion behavior - Ensure test cases cover new progress tracker functionality
- Remove superfluous 'index' parameter from updateStepStatus method - Update method signature and calls to updateStepStatus - Ensure waitForCompletion always returns a Promise<void>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add Event-Driven Validation Progress Tracker
This PR introduces a new event-driven validation progress tracker to provide users with real-time, step-by-step updates during the validation process. It also includes improvements to JSON-LD handling and updates the schema.org namespace from HTTPS to HTTP.
Changes
1. Add Step-by-Step Progress Tracker
ValidationProgressTrackerclass for real-time validation progressThis enhancement provides users with a more detailed and interactive view of the validation process, allowing them to see the progress and results of individual checks as they occur.
2. Add Event Emission for Validation Checks
emitCheckfunction to standardize event emission for validation checks3. Implement Event-Driven Progress Tracking Option
useEventsflag inValidatorOptionsindex.ts,psychds-validator.ts, andvalidate.tsto support event-based validationValidationProgressTrackerfor real-time progress updatesEventEmitterwhenuseEventsis trueThis change allows users to opt-in to event-driven validation, providing real-time progress updates and improved visibility into the validation process. The existing synchronous validation remains the default behavior.
4. Refactor Metadata Handling and Update Schema.org Namespace
context.ts5. Add CLI Option for Sequential Validation Progress Display
useEventsflag inValidatorOptionstype--useEventscommand-line option to the CLI parserparseOptionsfunction to includeuseEventsin returned options object6. Update Tests for Progress Tracker and HTTP Schema.org Namespace
options.test.tsto includeuseEventsflag in expected outputcontext.test.tsto use HTTP instead of HTTPS for schema.org namespacecontext.test.tsto reflect new JSON-LD expansion behaviorSummary
This PR significantly enhances the user experience of the validation process by providing real-time, step-by-step progress updates. It also improves the handling of JSON-LD and updates the schema.org namespace to use HTTP. The changes are backwards compatible, allowing users to opt-in to the new event-driven validation while maintaining the existing synchronous behavior as the default.