Skip to content

Commit 66fa635

Browse files
thomaspoignantthomaspoignant
authored
Adding the support to google cloud storage as data exporter (#231)
1 parent 8291f62 commit 66fa635

File tree

10 files changed

+485
-39
lines changed

10 files changed

+485
-39
lines changed

‎README.md

+24-23
Original file line numberDiff line numberDiff line change
@@ -117,17 +117,17 @@ ffclient.Init(ffclient.Config{
117117
```
118118
### Configuration fields
119119

120-
| Field | Description |
121-
|---|---|
122-
|`Retriever` | The configuration retriever you want to use to get your flag file.<br> *See [Store your flag file](https://thomaspoignant.github.io/go-feature-flag/latest/flag_file/) for the configuration details*.|
123-
|`Context` | *(optional)*<br>The context used by the retriever.<br />Default: `context.Background()`|
124-
|`DataExporter` | *(optional)*<br>DataExporter defines how to export data on how your flags are used.<br> *see [export data section](https://thomaspoignant.github.io/go-feature-flag/latest/data_collection/) for more details*.|
125-
|`FileFormat`| *(optional)*<br>Format of your configuration file. Available formats are `yaml`, `toml` and `json`, if you omit the field it will try to unmarshal the file as a `yaml` file.<br>Default: `YAML`|
126-
|`Logger` | *(optional)*<br>Logger used to log what `go-feature-flag` is doing.<br />If no logger is provided the module will not log anything.<br>Default: No log|
127-
|`Notifiers` | *(optional)*<br>List of notifiers to call when your flag file has been changed.<br> *See [notifiers section](https://thomaspoignant.github.io/go-feature-flag/latest/notifier/) for more details*.|
128-
|`PollingInterval` | *(optional)* Duration to wait before refreshing the flags.<br>The minimum polling interval is 1 second.<br>Default: 60 * time.Second|
129-
|`StartWithRetrieverError` | *(optional)*<br>If **true**, the SDK will start even if we did not get any flags from the retriever. It will serve only default values until the retriever returns the flags.<br>The init method will not return any error if the flag file is unreachable.<br>Default: **false**|
130-
|`Offline`| *(optional)* If **true**, the SDK will not try to retrieve the flag file and will not export any data. No notification will be send neither.<br>Default: false|
120+
| Field | Description |
121+
|---------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
122+
| `Retriever` | The configuration retriever you want to use to get your flag file.<br> *See [Store your flag file](https://thomaspoignant.github.io/go-feature-flag/latest/flag_file/) for the configuration details*. |
123+
| `Context` | *(optional)*<br>The context used by the retriever.<br />Default: `context.Background()` |
124+
| `DataExporter` | *(optional)*<br>DataExporter defines how to export data on how your flags are used.<br> *see [export data section](https://thomaspoignant.github.io/go-feature-flag/latest/data_collection/) for more details*. |
125+
| `FileFormat` | *(optional)*<br>Format of your configuration file. Available formats are `yaml`, `toml` and `json`, if you omit the field it will try to unmarshal the file as a `yaml` file.<br>Default: `YAML` |
126+
| `Logger` | *(optional)*<br>Logger used to log what `go-feature-flag` is doing.<br />If no logger is provided the module will not log anything.<br>Default: No log |
127+
| `Notifiers` | *(optional)*<br>List of notifiers to call when your flag file has been changed.<br> *See [notifiers section](https://thomaspoignant.github.io/go-feature-flag/latest/notifier/) for more details*. |
128+
| `PollingInterval` | *(optional)* Duration to wait before refreshing the flags.<br>The minimum polling interval is 1 second.<br>Default: 60 * time.Second |
129+
| `StartWithRetrieverError` | *(optional)*<br>If **true**, the SDK will start even if we did not get any flags from the retriever. It will serve only default values until the retriever returns the flags.<br>The init method will not return any error if the flag file is unreachable.<br>Default: **false** |
130+
| `Offline` | *(optional)* If **true**, the SDK will not try to retrieve the flag file and will not export any data. No notification will be send neither.<br>Default: false |
131131

132132
### Multiple configuration flag files
133133
`go-feature-flag` comes ready to use out of the box by calling the `Init` function and it will be available everywhere.
@@ -251,18 +251,18 @@ version = 12.0
251251
</details>
252252

253253

254-
| Field | Description |
255-
|:---:|---|
256-
| **flag-key** | Name of your flag.<br> It must be unique.<br>*On the example the flag keys are **`test-flag`** and **`test-flag2`**.*|
257-
| `true` | Value returned by the flag if the rule is evaluated to true and the user is in the active percentage.|
258-
| `false`| Value returned by the flag if the rule is evaluated to true and the user is **not** in the active percentage.|
259-
| `default` |Value returned by the flag if the rule is evaluated to false.|
260-
| `percentage` |*(optional)*<br>Percentage of users who should be affected by the flag.<br>**Default: 0**<br><br>The percentage is computed by calculating a hash of the user key *(100000 variations)*, it means that you can have 3 numbers after the comma.|
261-
| `rule` |*(optional)*<br>Condition to determine on which user the flag should be applied.<br>Rule format is described in the <a href="#rule-format">rule format section</a>.<br>**If no rule is set, the flag applies to all users *(percentage still apply)*.**|
262-
| `disable` |*(optional)*<br>True if the flag is disabled.<br>**Default: `false`**|
263-
| `trackEvents` |*(optional)*<br>False if you don't want to export the data in your data exporter.<br>**Default: `true`**|
264-
| `version` |*(optional)*<br>The version is the version of your flag.<br>This number is used to display the information in the notifiers and data collection, you have to update it your self.<br>**Default: 0**|
265-
| `rollout` |*(optional)*<br><code>rollout</code> contains a specific rollout strategy you want to use.<br>**See [rollout section](https://thomaspoignant.github.io/go-feature-flag/latest/rollout/) for more details.**|
254+
| Field | Description |
255+
|:-------------:|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
256+
| **flag-key** | Name of your flag.<br> It must be unique.<br>*On the example the flag keys are **`test-flag`** and **`test-flag2`**.* |
257+
| `true` | Value returned by the flag if the rule is evaluated to true and the user is in the active percentage. |
258+
| `false` | Value returned by the flag if the rule is evaluated to true and the user is **not** in the active percentage. |
259+
| `default` | Value returned by the flag if the rule is evaluated to false. |
260+
| `percentage` | *(optional)*<br>Percentage of users who should be affected by the flag.<br>**Default: 0**<br><br>The percentage is computed by calculating a hash of the user key *(100000 variations)*, it means that you can have 3 numbers after the comma. |
261+
| `rule` | *(optional)*<br>Condition to determine on which user the flag should be applied.<br>Rule format is described in the <a href="#rule-format">rule format section</a>.<br>**If no rule is set, the flag applies to all users *(percentage still apply)*.** |
262+
| `disable` | *(optional)*<br>True if the flag is disabled.<br>**Default: `false`** |
263+
| `trackEvents` | *(optional)*<br>False if you don't want to export the data in your data exporter.<br>**Default: `true`** |
264+
| `version` | *(optional)*<br>The version is the version of your flag.<br>This number is used to display the information in the notifiers and data collection, you have to update it your self.<br>**Default: 0** |
265+
| `rollout` | *(optional)*<br><code>rollout</code> contains a specific rollout strategy you want to use.<br>**See [rollout section](https://thomaspoignant.github.io/go-feature-flag/latest/rollout/) for more details.** |
266266

267267
## Rule format
268268
The rule format is based on the [`nikunjy/rules`](https://github.com/nikunjy/rules) library.
@@ -385,6 +385,7 @@ It collects all the variations events and can save these events on several locat
385385
- [File](https://thomaspoignant.github.io/go-feature-flag/latest/data_collection/file/) *- create local files with the variation usages.*
386386
- [Log](https://thomaspoignant.github.io/go-feature-flag/latest/data_collection/log/) *- use your logger to write the variation usages.*
387387
- [S3](https://thomaspoignant.github.io/go-feature-flag/latest/data_collection/s3/) *- export your variation usages to S3.*
388+
- [Google Cloud Storage](https://thomaspoignant.github.io/go-feature-flag/latest/flag_file/data_collection/google_cloud_storage) *- export your variation usages to Google Cloud Storage.*
388389
- [Webhook](https://thomaspoignant.github.io/go-feature-flag/latest/data_collection/webhook/) *- export your variation usages by calling a webhook.*
389390

390391
Currently, we are supporting only feature events.

‎docs/data_collection/google_cloud_storage.md

+40
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Google Cloud Storage Exporter
2+
3+
The **Google Cloud Storage exporter** will collect the data and create a new file in a specific folder everytime we send the data.
4+
5+
Everytime the `FlushInterval` or `MaxEventInMemory` is reached a new file will be added to S3.
6+
7+
!!! Info
8+
If for some reason the Google Cloud Storage upload failed, we will keep the data in memory and retry to add the next time we reach `FlushInterval` or `MaxEventInMemory`.
9+
10+
Check this [complete example](https://github.com/thomaspoignant/go-feature-flag/tree/main/examples/data_export_googlecloudstorage) to see how to export the data in S3.
11+
12+
## Configuration example
13+
```go linenums="1"
14+
ffclient.Config{
15+
// ...
16+
DataExporter: ffclient.DataExporter{
17+
// ...
18+
Exporter: &ffexporter.GoogleCloudStorage{
19+
Bucket: "test-goff",
20+
Format: "json",
21+
Path: "yourPath",
22+
Filename: "flag-variation-{{ .Timestamp}}.{{ .Format}}",
23+
Options: []option.ClientOption{}, // Your google cloud SDK options
24+
},
25+
},
26+
// ...
27+
}
28+
```
29+
30+
## Configuration fields
31+
| Field | Description |
32+
|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
33+
| `Bucket ` | Name of your Google Cloud Storage Bucket. |
34+
| `CsvTemplate` | *(optional)* CsvTemplate is used if your output format is CSV. This field will be ignored if you are using another format than CSV. You can decide which fields you want in your CSV line with a go-template syntax, please check [internal/exporter/feature_event.go](https://github.com/thomaspoignant/go-feature-flag/blob/main/internal/exporter/feature_event.go) to see what are the fields available.<br>**Default:** `{{ .Kind}};{{ .ContextKind}};{{ .UserKey}};{{ .CreationDate}};{{ .Key}};{{ .Variation}};{{ .Value}};{{ .Default}}\n` |
35+
| `Filename` | *(optional)* Filename is the name of your output file. You can use a templated config to define the name of your exported files.<br>Available replacement are `{{ .Hostname}}`, `{{ .Timestamp}`} and `{{ .Format}}`<br>Default: `flag-variation-{{ .Hostname}}-{{ .Timestamp}}.{{ .Format}}` |
36+
| `Format` | *(optional)* Format is the output format you want in your exported file. Available format are **`JSON`** and **`CSV`**. *(Default: `JSON`)* |
37+
| `Options` | *(optional)* An instance of `option.ClientOption` that configures your access to Google Cloud. <br> Check [this documentation for more info](https://cloud.google.com/docs/authentication). |
38+
| `Path ` | *(optional)* The location of the directory in your bucket. |
39+
40+
Check the [godoc for full details](https://pkg.go.dev/github.com/thomaspoignant/go-feature-flag/ffexporter#GoogleCloudStorage).

0 commit comments

Comments
 (0)