Skip to content

Commit 6465042

Browse files
MisterGCMisterGC
authored
Merge pull request #9 from ndsev/0.4.0
Version 0.4.0
2 parents e9165c0 + 88bf0c6 commit 6465042

14 files changed

+484
-88
lines changed

‎.github/workflows/ci.yml

+67
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
name: zs-yaml CI/CD
2+
on:
3+
push:
4+
branches:
5+
- main
6+
tags:
7+
- 'v*'
8+
pull_request:
9+
branches:
10+
- main
11+
12+
jobs:
13+
test-build-and-deploy:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v3
17+
- name: Set up Python
18+
uses: actions/setup-python@v4
19+
with:
20+
python-version: '3.12'
21+
- name: Install dependencies
22+
run: |
23+
python -m pip install --upgrade pip
24+
pip install setuptools wheel twine
25+
- name: Install zs-yaml in dev mode
26+
run: |
27+
pip install -e .
28+
- name: Generate schema API for team example
29+
run: |
30+
cd examples/team
31+
zserio team.zs -withTypeInfoCode -python zs_gen_api
32+
echo "PYTHONPATH=$PYTHONPATH:$PWD/zs_gen_api" >> $GITHUB_ENV
33+
- name: Test team example transformation
34+
run: |
35+
cd examples/team
36+
zs-yaml team1.yaml team1_test_out.bin
37+
if [ $? -eq 0 ]; then
38+
echo "Transformation successful"
39+
else
40+
echo "Transformation failed"
41+
exit 1
42+
fi
43+
- name: Set version
44+
run: |
45+
if [[ $GITHUB_REF == refs/tags/v* ]]; then
46+
VERSION=${GITHUB_REF#refs/tags/v}
47+
elif [[ $GITHUB_REF == refs/heads/main ]]; then
48+
VERSION=$(python setup.py --version)
49+
else
50+
VERSION=$(python setup.py --version).dev0
51+
fi
52+
echo "VERSION=${VERSION}" >> $GITHUB_ENV
53+
echo "Extracted version: ${VERSION}"
54+
sed -i "s/__version__ = .*/__version__ = '${VERSION}'/" zs_yaml/_version.py
55+
- name: Build package
56+
run: |
57+
python setup.py sdist bdist_wheel
58+
- name: Check distribution
59+
run: |
60+
twine check dist/*
61+
- name: Publish to PyPI
62+
if: startsWith(github.ref, 'refs/tags/v')
63+
env:
64+
TWINE_USERNAME: __token__
65+
TWINE_PASSWORD: ${{ secrets.KE_PYPI_TOKEN }}
66+
run: |
67+
twine upload dist/*

‎README.md

+95-30
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,91 @@
1010
- **Metadata Inclusion**: Automatically includes metadata in the YAML, eliminating the need for users to manually identify the correct type when importing JSON data, ensuring seamless (de-)serialization.
1111
- **Custom Transformations**: Allows for hooking in custom transformations so that users can work with familiar formats (e.g., dates or coordinate representations) instead of thinking in unfamiliar formats.
1212

13+
## Design Principles
14+
15+
1. **Transparency Over Magic**:
16+
- Prioritize clear and understandable processes. For example, users can fully render the YAML to see the actual data conforming to the underlying zserio schema.
17+
- This approach avoids black-box conversions, simplifying debugging and ensuring user confidence in the data.
18+
19+
2. **Accessibility and Simplicity**:
20+
- Make the tool easy to use and understand, even for beginners.
21+
- Features are designed with simplicity in mind. For instance, we use string-only templates as one way to keep things straightforward.
22+
23+
3. **Performance for Trusted Sources**:
24+
- Optimize for performance, assuming trusted YAML sources.
25+
- Faster processing is crucial for rapid iterations and maintaining workflow.
26+
27+
4. **Flexibility Within Simplicity**:
28+
- While maintaining a simple core, provide powerful features like YAML imports, built-in and custom transformations, and basic templating.
29+
- This balance allows for adaptability to various use cases without compromising ease of use.
30+
31+
## Installation
32+
33+
Install `zs-yaml` using pip:
34+
35+
```bash
36+
python -m pip install --upgrade zs-yaml
37+
```
38+
39+
## Usage
40+
41+
The main entry point for the application is `zs-yaml`. It accepts arguments for specifying the input and output file paths. You can run the application as follows:
42+
43+
```bash
44+
zs-yaml input.yaml output.bin
45+
zs-yaml input.bin output.yaml
46+
```
47+
48+
### Notes
49+
50+
- You have to use the exact same order of fields in the YAML as defined by the zserio schema, because zserio expects this.
51+
- When converting from binary to YAML, the target YAML file must already exist and contain the necessary metadata.
52+
- The minimal metadata content in the target YAML file should be:
53+
54+
```yaml
55+
_meta:
56+
schema_module: <module_name>
57+
schema_type: <type_name>
58+
```
59+
60+
### Initialization Arguments
61+
62+
Some Zserio types require additional arguments during initialization, either when deserializing from binary or when creating objects from JSON. To support these types, you can specify initialization arguments in the `_meta` section of your YAML file:
63+
64+
```yaml
65+
_meta:
66+
schema_module: <module_name>
67+
schema_type: <type_name>
68+
initialization_args:
69+
- <arg1>
70+
- <arg2>
71+
# ... more arguments as needed
72+
```
73+
74+
**Hint:** At the moment only plain values are supported, although zserio supports also compound values as args.
75+
Support for these may be added in the future.
76+
77+
These arguments will be passed to the appropriate Zserio functions:
78+
- `zserio.deserialize_from_file()` when converting from binary to YAML
79+
- `zserio.from_json_file()` when converting from YAML to binary
80+
81+
For example:
82+
83+
```yaml
84+
_meta:
85+
schema_module: my_schema
86+
schema_type: MyType
87+
initialization_args:
88+
- 0xAB
89+
- 42
90+
91+
# ... rest of your YAML data
92+
```
93+
94+
In this example, `0xAB` and `42` will be passed as additional arguments to the initialization functions.
95+
96+
This feature ensures that types requiring additional initialization parameters can be properly handled in both directions of conversion (YAML to binary and binary to YAML).
97+
1398
## Example
1499

15100
### Zserio Schema Module Creation
@@ -60,8 +145,9 @@ Using **zs-yaml**, you can define the same data in a more human-readable YAML fo
60145
```yaml
61146
# 1) Metadata is used to specify the type needed for
62147
# (de-)serialization and custom transform functions
63-
# 2) User are free to use their preferred date format
148+
# 2) Users are free to use their preferred date format
64149
# for the birth data as the a normalization function
150+
# (defined in the referenced `transformations.py`)
65151
# get invoked.
66152
# 3) Yaml allows avoiding clutter and adding comments
67153
# like this one :)
@@ -114,39 +200,18 @@ After you have installed `zs-yaml`, call `zs-yaml` to convert your YAML file to
114200
zs-yaml person.yaml person.bin
115201
```
116202

117-
## Installation
118-
119-
Currently, the Python package is only available at test.pypi. Once the official release is available,
120-
you can install the package without specifying an index URL.
121-
122-
```bash
123-
python -m pip install --upgrade zs-yaml
124-
```
125-
126-
## Usage
127-
128-
The main entry point for the application is `main.py`. It accepts arguments for specifying the input and output file paths. You can run the application as follows:
129-
130-
```bash
131-
zs-yaml input.yaml output.bin
132-
zs-yaml input.bin output.yaml
133-
```
134-
135-
### Notes
203+
## Built-in Transformations
136204

137-
- You have to use the exact same order of field in the yaml as defined by the zserio schema, because zserio expects this
138-
- When converting from binary to YAML, the target YAML file must already exist and contain the necessary metadata.
139-
- The minimal metadata content in the target YAML file should be:
205+
zs-yaml comes with several built-in transformation functions that can be used in your YAML files. Here's a brief overview of the available functions:
140206

141-
```yaml
142-
_meta:
143-
schema_module: <module_name>
144-
schema_type: <type_name>
145-
```
207+
- `insert_yaml_as_extern`: Includes external YAML content by transforming it to JSON and using zserio.
208+
- `insert_yaml`: Inserts YAML content directly from an external file.
209+
- `repeat_node`: Repeats a specific node a specified number of times.
210+
- `extract_extern_as_yaml`: Extracts binary data and saves it as an external YAML file.
146211

147-
### Future Considerations
212+
For more detailed information about these functions and their usage, please refer to the [built_in_transformations.py](https://github.com/ndsev/zs_yaml/blob/main/zs_yaml/built_in_transformations.py) source file.
148213

149-
In the future, we may consider using YAML tags for a more integrated approach to handling transformations directly within the YAML files, simplifying syntax and enhancing readability.
214+
Note: We plan to implement automatic source documentation generation in a future release, which will provide more comprehensive information about these functions and their parameters.
150215

151216
## Project Structure
152217

‎examples/team/addresses.yaml

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
2+
- city_only: "New York"
3+
detailed:
4+
street: "123 Main St"
5+
city: "New York"
6+
country: "USA"
7+
zipCode: 10001
8+
9+
- city_only: "San Francisco"
10+
detailed:
11+
street: "456 Elm St"
12+
city: "San Francisco"
13+
country: "USA"
14+
zipCode: 94102

‎examples/team/custom_transformations.py

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
from datetime import datetime
2+
3+
4+
def calculate_age(transformer, birthdate):
5+
born = datetime.strptime(birthdate, "%Y-%m-%d")
6+
today = datetime.today()
7+
return today.year - born.year - ((today.month, today.day) < (born.month, born.day))

‎examples/team/hobbies.yaml

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
common:
2+
- "Reading"
3+
- "Traveling"
4+
- "Photography"
5+
6+
jane_smith:
7+
- "Hiking"
8+
- "Cooking"
9+
- "Painting"

‎examples/team/skills.yaml

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- name: "Python"
2+
level: 9
3+
- name: "JavaScript"
4+
level: 8
5+
- name: "Database Management"
6+
level: 7

‎examples/team/team.zs

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
package team;
2+
3+
struct Address {
4+
string street;
5+
string city;
6+
string country;
7+
uint32 zipCode;
8+
};
9+
10+
struct Experience {
11+
string company;
12+
string position;
13+
uint16 yearsWorked;
14+
};
15+
16+
struct Skill {
17+
string name;
18+
uint8 level;
19+
};
20+
21+
struct Person {
22+
string name;
23+
uint32 age;
24+
Address address;
25+
Experience workExperience[];
26+
Skill skills[];
27+
string hobbies[];
28+
string bio;
29+
};
30+
31+
struct Team {
32+
string name;
33+
Person members[];
34+
};

‎examples/team/team1.yaml

+61
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
_meta:
2+
schema_module: team.api
3+
schema_type: Team
4+
transformation_module: "./custom_transformations.py"
5+
6+
name: "Dream Team"
7+
8+
members:
9+
- name: "John Doe"
10+
age:
11+
_f: calculate_age
12+
_a: "1990-05-15"
13+
address:
14+
_f: insert_yaml
15+
_a:
16+
file: "addresses.yaml"
17+
node_path: "[0].detailed"
18+
workExperience:
19+
_f: insert_yaml
20+
_a:
21+
file: "work_experience.yaml"
22+
node_path: "john_doe"
23+
skills:
24+
_f: repeat_node
25+
_a:
26+
count: 4
27+
node:
28+
_f: insert_yaml
29+
_a:
30+
file: "skills.yaml"
31+
node_path: "[0]"
32+
hobbies:
33+
_f: insert_yaml
34+
_a:
35+
file: "hobbies.yaml"
36+
node_path: "common"
37+
bio: "Just some bio."
38+
39+
- name: "Jane Smith"
40+
age:
41+
_f: calculate_age
42+
_a: "1988-09-23"
43+
address:
44+
_f: insert_yaml
45+
_a:
46+
file: "addresses.yaml"
47+
node_path: "[1].detailed"
48+
workExperience:
49+
_f: insert_yaml
50+
_a:
51+
file: "work_experience.yaml"
52+
node_path: "jane_smith"
53+
skills:
54+
_f: insert_yaml
55+
_a: "skills.yaml"
56+
hobbies:
57+
_f: insert_yaml
58+
_a:
59+
file: "hobbies.yaml"
60+
node_path: "jane_smith"
61+
bio: "Another bio."

‎examples/team/work_experience.yaml

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
john_doe:
2+
- company: "Tech Corp"
3+
position: "Senior Developer"
4+
yearsWorked: 5
5+
- company: "Startup Inc"
6+
position: "Junior Developer"
7+
yearsWorked: 2
8+
9+
jane_smith:
10+
- company: "Big Enterprise"
11+
position: "Project Manager"
12+
yearsWorked: 6
13+
- company: "Small Business"
14+
position: "Business Analyst"
15+
yearsWorked: 4

‎requirements.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
PyYAML~=6.0
2-
zserio~=2.12
2+
zserio~=2.14.1

‎zs_yaml/_version.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
1-
__commit_id__ = 'ebfe593'
2-
__version__ = '0.3.0'
1+
__commit_id__ = '7217bfe'
2+
__version__ = '0.4.0'

0 commit comments

Comments
 (0)