Logging operator for Kubernetes based on Fluentd and Fluent-bit. For more details please follow up with this post.
Logging-operator is a core part of the Pipeline platform, a Cloud Native application and devops platform that natively supports multi- and hybrid-cloud deployments with multiple authentication backends. Check out the developer beta:
This operator helps you to pack together logging information with your applications. With the help of Custom Resource Definition you can describe the behaviour of your application within its charts. The operator does the rest.
The following steps set up an example configuration for sending nginx logs to S3.
$ helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
$ helm repo update$ helm install banzaicloud-stable/logging-operator$ helm install \
--set bucketName='<Mybucket>' \
--set region='<S3_REGION>' \
--set endpoint='<S3_ENDPOINT>' \
--set awsCredentialsAccess.enabled=true \
--set awsCredentialsAccess.secret.awsAccessValue='<AWS_ACCESS_KEY_ID>' \
--set awsCredentialsAccess.secret.awsSecretValue='<AWS_SECRET_ACCESS_KEY>' \
banzaicloud-stable/s3-outputThere is no need to encode base64 these values.
$ helm install banzaicloud-stable/nginx-logging-demo# Create all the CRDs used by the Operator
kubectl create -f deploy/crds/logging_v1alpha1_plugin_crd.yaml
kubectl create -f deploy/crds/logging_v1alpha1_fluentbit_crd.yaml
kubectl create -f deploy/crds/logging_v1alpha1_fluentd_crd.yaml
# If RBAC enabled create the required resources
kubectl create -f deploy/clusterrole.yaml
kubectl create -f deploy/clusterrole_binding.yaml
kubectl create -f deploy/service_account.yaml
# Create the Operator
kubectl create -f deploy/operator.yaml
# Create the fluent-bit daemonset by submiting a fluent-bit CR
kubectl create -f deploy/crds/logging_v1alpha1_fluentbit_cr.yaml
# Create the fluentd deployment by submitting a fluentd CR
kubectl create -f deploy/crds/logging_v1alpha1_fluentd_cr.yaml
Create a manifest file for the AWS access key:
apiVersion: v1
kind: Secret
metadata:
name: loggings3
type: Opaque
data:
awsAccessKeyId: <base64encoded>
awsSecretAccesKey: <base64encoded>
Submit the secret with kubectl:
kubectl apply -f secret.yaml
Create a manifest that defines that you want to parse the nginx logs with the specified regular expressions on the standard output of pods with the app: nginx label, and store them in the given S3 bucket.
apiVersion: "logging.banzaicloud.com/v1alpha1"
kind: "Plugin"
metadata:
name: "nginx-logging"
labels:
release: test
spec:
input:
label:
app: nginx
filter:
- type: parser
name: parser-nginx
parameters:
- name: format
value: '/^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"(?:\s+(?<http_x_forwarded_for>[^ ]+))?)?$/'
- name: timeFormat
value: "%d/%b/%Y:%H:%M:%S %z"
output:
- type: s3
name: outputS3
parameters:
- name: aws_key_id
valueFrom:
secretKeyRef:
name: loggings3
key: awsAccessKeyId
- name: aws_sec_key
valueFrom:
secretKeyRef:
name: loggings3
key: awsSecretAccesKey
- name: s3_bucket
value: logging-bucket
- name: s3_region
value: ap-northeast-1
- name: s3_endpoint
value: https://s3.amazonaws.com
$ helm repo add es-operator https://raw.githubusercontent.com/upmc-enterprises/elasticsearch-operator/master/charts/
$ helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
$ helm repo update$ helm install --name elasticsearch-operator es-operator/elasticsearch-operator --set rbac.enabled=True
$ helm install --name elasticsearch es-operator/elasticsearch --set kibana.enabled=True --set cerebro.enabled=True
$ helm install --name loggingo banzaicloud-stable/logging-operator$ helm install banzaicloud-stable/nginx-logging-es-demo$ kubectl port-forward svc/cerebro-elasticsearch-cluster 9001:80
$ kubectl port-forward svc/kibana-elasticsearch-cluster 5601:80
If you encounter any problems that the documentation does not address, please file an issue or talk to us on the Banzai Cloud Slack channel #logging-operator.
If you find this project useful here's how you can help:
- Send a pull request with your new features and bug fixes
- Help new users with issues they may encounter
- Support the development of this project and star this repo!
Copyright (c) 2017-2019 Banzai Cloud, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.