Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Next release conflict fixed #1996

Merged
merged 52 commits into from
Jun 1, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
06f2309
DROTH-3228 add missing webhook
sasuolanderSito May 3, 2022
207fb18
Merge pull request #1961 from finnishtransportagency/DROTH-3228_fix_p…
sasuolanderSito May 3, 2022
471bb2b
DROTH-3214 Batch system divided for prod and dev account. Added Cloud…
anttiahopeltositowise May 6, 2022
dd60d3d
DROTH-3214 Added instructions for creating stacks
anttiahopeltositowise May 6, 2022
e733faa
DROTH-3214 Divided pipeline into two. Added tags, added instructions …
anttiahopeltositowise May 11, 2022
dfd1048
DROTH-3214 Added owner and project tags, added event rule for Refresh…
anttiahopeltositowise May 11, 2022
4111747
DROTH-3214 Added missing ApplicationName parameter and Tag
anttiahopeltositowise May 11, 2022
0818834
DROTH-3214 Changed parameter names
anttiahopeltositowise May 12, 2022
1e6b50b
DROTH-3214 Fixed yaml syntax errors in lambda pipeline
anttiahopeltositowise May 12, 2022
8524f81
DROTH-3214 Fixed yaml syntax errors in batch system, added SecurityGr…
anttiahopeltositowise May 16, 2022
19cd6d9
DROTH-3214 Fixed pipeline permissions
anttiahopeltositowise May 16, 2022
0b6d370
DROTH-3214 New function name for lambda
anttiahopeltositowise May 16, 2022
331bb14
DROTH-3214 fix prod pipeline YAML, add artifact store
anttiahopeltositowise May 16, 2022
dd38ad3
DROTH-3214 add parameters to prod deployment bucket, fix YAML
anttiahopeltositowise May 16, 2022
1121eb4
Merge pull request #1962 from finnishtransportagency/DROTH-3214_run_p…
anttiahopeltositowise May 17, 2022
0468daa
DROTH-3214 added --parameters to S3 creation guide, renamed parameter
anttiahopeltositowise May 17, 2022
4671d28
DROTH-3214 fix params
anttiahopeltositowise May 17, 2022
e7c1435
Merge pull request #1966 from finnishtransportagency/DROTH-3214_run_p…
anttiahopeltositowise May 17, 2022
1839b58
DROTH-3259 enable caching QA job definition
anttiahopeltositowise May 17, 2022
fc6933a
DROTH-3259 also enable caching prod job definition
anttiahopeltositowise May 17, 2022
7e217a4
Merge pull request #1968 from finnishtransportagency/DROTH_3259_enabl…
anttiahopeltositowise May 17, 2022
6056628
DROTH-3198 Out of memory fix
anttiahopeltositowise May 17, 2022
dfc4c2b
DROTH-3198 Fixed roleArn in job definitions to correct new arn. Fixed…
anttiahopeltositowise May 18, 2022
9cb33da
DROTH-3198 included fix in batchSystem.yaml
anttiahopeltositowise May 18, 2022
489c68b
Merge pull request #1971 from finnishtransportagency/DROTH-3214_run_p…
anttiahopeltositowise May 18, 2022
7719458
DROTH-3198 added exit() for refreshCache
anttiahopeltositowise May 18, 2022
cac7d2b
DROTH-3258 add new endpoint and needed properties
sasuolanderSito May 18, 2022
1db791b
DROTH-3258 update cloudformations
sasuolanderSito May 18, 2022
8961ce0
DROTH-3258 update env.properties
sasuolanderSito May 18, 2022
e8e77b6
DROTH-3258 update readme
sasuolanderSito May 18, 2022
f94ca64
DROTH-3258 update readme
sasuolanderSito May 18, 2022
5a66df3
DROTH-3214 Fixed JobDefinition
anttiahopeltositowise May 19, 2022
e271f4a
Merge pull request #1975 from finnishtransportagency/DROTH-3214_run_p…
anttiahopeltositowise May 19, 2022
88e866d
DROTH-3214 batch fail topic and event rule moved to own template, no …
anttiahopeltositowise May 19, 2022
b4e0824
DROTH-3214 moved sns files to own directory
anttiahopeltositowise May 23, 2022
fd45021
DROTH-3214 renamed files for future
anttiahopeltositowise May 23, 2022
3ce9246
DROTH-3258 update parameter.sh
sasuolanderSito May 23, 2022
fc7f117
DROTH-3214 renamed parameter
anttiahopeltositowise May 23, 2022
fc90259
DROTH-3258 Merge branch 'master' into DROTH-3258_add_basicAuth_vvh
sasuolanderSito May 23, 2022
0658aa7
DROTH-3258 fix account id
sasuolanderSito May 23, 2022
271affc
DROTH-3214 fixed Input transformer, added instructions for creating S…
anttiahopeltositowise May 23, 2022
7d72052
Merge pull request #1976 from finnishtransportagency/DROTH-3214_run_p…
anttiahopeltositowise May 23, 2022
2f2e07c
Merge pull request #1980 from finnishtransportagency/DROTH-3198_Updat…
anttiahopeltositowise May 24, 2022
c4373ee
Merge pull request #1974 from finnishtransportagency/DROTH-3258_add_b…
sasuolanderSito May 24, 2022
a235d20
DROTH-3198 Fixed bug with duplicate changes to link attributes
anttiahopeltositowise May 25, 2022
1419d8b
DROTH-3198 Added try/catch clause for updating road link attributes
anttiahopeltositowise May 30, 2022
17fb4f6
Merge pull request #1986 from finnishtransportagency/DROTH-3198_Updat…
anttiahopeltositowise May 30, 2022
decee5a
Merge pull request #1987 from finnishtransportagency/NextRelease
anttiahopeltositowise May 31, 2022
345e62d
pull image with test tag, default was latest
sasuolanderSito May 31, 2022
aacc6a3
Merge pull request #1988 from finnishtransportagency/fixProdpipeline
sasuolanderSito May 31, 2022
81cc8fb
Merge branch 'NextRelease' into master_into_nextRelease
anttiahopeltositowise Jun 1, 2022
f03a152
Merge branch 'master_into_nextRelease' into NextRelease_conflict_fixed
anttiahopeltositowise Jun 1, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,9 +85,8 @@ API-palvelimen saa käyntiin kehitysmoodiin seuraavalla sbt komennolla:
./sbt '~;container:start; container:reload /'
```

Salasanat voidaan syöttää ympäristömuuttujina myös.
Parametri voidaan asettaa Intellij SBT Configuration Environment Variable.
Nämä voidaan syöttään myös ympäristömuuttujina.
Ympäristömuuttuja parametri voidaan asettaa Intellij SBT Configuration Environment Variable avulla.
Nämä voidaan syöttään myös ympäristömuuttujina:
```
viiteRestApiEndPoint=url
viite.apikey=insertapikey
Expand All @@ -102,6 +101,7 @@ ses.password=sespassword
bonecp.jdbcUrl=kantaurl
bonecp.username=kantakäyttäjä
bonecp.password=kantasalasana
vvhRest.password=insertpassword
```
Windowsissa toimii komento:
```
Expand Down
14 changes: 11 additions & 3 deletions aws/cloud-formation/batchSystem/DEVbatchJobDefinition.json
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"value": "4"
}
],
"executionRoleArn": "arn:aws:iam::475079312496:role/batchSystem-BatchTaskRole-GSUD7E3H8KV4",
"executionRoleArn": "arn:aws:iam::475079312496:role/DEV-batchSystem-BatchTaskRole-VGL42N9TANTI",
"environment": [
{
"name": "containerCPU",
Expand Down Expand Up @@ -50,11 +50,15 @@
},
{
"name": "vvhServiceHost",
"value": "haproxy.vayla.fi"
"value": "api.vayla.fi"
},
{
"name": "vvhRestApiEndPoint",
"value": "https://haproxy.vayla.fi:2027/vvhdata/"
"value": "https://api.vayla.fi/vvhdata/"
},
{
"name": "vvhRest.username",
"value": "svc_vvh_digiroad"
},
{
"name": "vvhRoadlink.frozen",
Expand Down Expand Up @@ -132,6 +136,10 @@
"name": "bonecp.password",
"valueFrom": "arn:aws:ssm:eu-west-1:475079312496:parameter/dev/bonecp.password"
},
{
"name": "vvhRest.password",
"valueFrom": "arn:aws:ssm:eu-west-1:475079312496:parameter/dev/vvhRest.password"
},
{
"name": "oag.password",
"valueFrom": "arn:aws:ssm:eu-west-1:475079312496:parameter/dev/authentication.oag.basic.password"
Expand Down
27 changes: 18 additions & 9 deletions aws/cloud-formation/batchSystem/ProdBatchJobDefinition.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"FARGATE"
],
"containerProperties": {
"image": "475079312496.dkr.ecr.eu-west-1.amazonaws.com/digiroad2:prod",
"image": "920408837790.dkr.ecr.eu-west-1.amazonaws.com/digiroad2:prod",
"resourceRequirements" : [
{
"type": "MEMORY",
Expand All @@ -22,7 +22,7 @@
"value": "4"
}
],
"executionRoleArn": "arn:aws:iam::475079312496:role/batchSystem-BatchTaskRole-GSUD7E3H8KV4",
"executionRoleArn": "arn:aws:iam::920408837790:role/digiroad-batch-system-BatchTaskRole-NTGCBVRU7CCR",
"environment": [
{
"name": "containerCPU",
Expand Down Expand Up @@ -50,11 +50,15 @@
},
{
"name": "vvhServiceHost",
"value": "haproxy.vayla.fi"
"value": "api.vayla.fi"
},
{
"name": "vvhRestApiEndPoint",
"value": "https://haproxy.vayla.fi:2027/vvhdata/"
"value": "https://api.vayla.fi/vvhdata/"
},
{
"name": "vvhRest.username",
"value": "svc_vvh_digiroad"
},
{
"name": "vvhRoadlink.frozen",
Expand Down Expand Up @@ -106,7 +110,7 @@
},
{
"name": "caching",
"value": "false"
"value": "true"
},
{
"name": "cacheHostname",
Expand All @@ -130,19 +134,24 @@
"secrets": [
{
"name": "bonecp.password",
"valueFrom": "arn:aws:ssm:eu-west-1:475079312496:parameter/prod/bonecp.password"
"valueFrom": "arn:aws:ssm:eu-west-1:920408837790:parameter/prod/bonecp.password"
},
{
"name": "vvhRest.password",
"valueFrom": "arn:aws:ssm:eu-west-1:920408837790:parameter/prod/vvhRest.password"
},
{
"name": "oag.password",
"valueFrom": "arn:aws:ssm:eu-west-1:475079312496:parameter/prod/authentication.oag.basic.password"
"valueFrom": "arn:aws:ssm:eu-west-1:920408837790:parameter/prod/authentication.oag.basic.password"
},

{
"name": "viite.apikey",
"valueFrom": "arn:aws:ssm:eu-west-1:475079312496:parameter/prod/apikey/viite"
"valueFrom": "arn:aws:ssm:eu-west-1:920408837790:parameter/prod/apikey/viite"
},
{
"name": "vkm.apikey",
"valueFrom": "arn:aws:ssm:eu-west-1:475079312496:parameter/prod/apikey/viitekehysmuunnin"
"valueFrom": "arn:aws:ssm:eu-west-1:920408837790:parameter/prod/apikey/viitekehysmuunnin"
}
]
}
Expand Down
16 changes: 12 additions & 4 deletions aws/cloud-formation/batchSystem/QAbatchJobDefinition.json
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"value": "4"
}
],
"executionRoleArn": "arn:aws:iam::475079312496:role/batchSystem-BatchTaskRole-GSUD7E3H8KV4",
"executionRoleArn": "arn:aws:iam::475079312496:role/QA-batchSystem-BatchTaskRole-1QO0VDAJOYS45",
"environment": [
{
"name": "containerCPU",
Expand Down Expand Up @@ -50,11 +50,15 @@
},
{
"name": "vvhServiceHost",
"value": "haproxy.vayla.fi"
"value": "api.vayla.fi"
},
{
"name": "vvhRestApiEndPoint",
"value": "https://haproxy.vayla.fi:2027/vvhdata/"
"value": "https://api.vayla.fi/vvhdata/"
},
{
"name": "vvhRest.username",
"value": "svc_vvh_digiroad"
},
{
"name": "vvhRoadlink.frozen",
Expand Down Expand Up @@ -106,7 +110,7 @@
},
{
"name": "caching",
"value": "false"
"value": "true"
},
{
"name": "cacheHostname",
Expand All @@ -132,6 +136,10 @@
"name": "bonecp.password",
"valueFrom": "arn:aws:ssm:eu-west-1:475079312496:parameter/qa/bonecp.password"
},
{
"name": "vvhRest.password",
"valueFrom": "arn:aws:ssm:eu-west-1:475079312496:parameter/qa/vvhRest.password"
},
{
"name": "oag.password",
"valueFrom": "arn:aws:ssm:eu-west-1:475079312496:parameter/qa/authentication.oag.basic.password"
Expand Down
26 changes: 0 additions & 26 deletions aws/cloud-formation/batchSystem/batch-parameter.json

This file was deleted.

82 changes: 82 additions & 0 deletions aws/cloud-formation/batchSystem/batchLambda/batchLambda.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
BucketName:
Description: Name of S3 bucket for lambda code
Type: String
S3ObjectKey:
Description: Key of lambda deployment object
Type: String
EnvironmentName:
Description: Name of AWS account dev or prod
Type: String
Owner:
Description: "Owner of the of application, that these resources are created for. Used when tagging the resources"
Type: String
Project:
Description: "Project of the of application, that these resources are created for. Used when tagging the resources"
Type: String
ApplicationName:
Description: Name of the application (no whitespace or special characters)
Type: String

Resources:
BatchLambda:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref BucketName
S3Key: !Ref S3ObjectKey
Tags:
- Key: Name
Value: !Join [ '-', [ !Ref EnvironmentName, !Ref ApplicationName, 'batchLambda' ] ]
- Key: Environment
Value: !Ref EnvironmentName
- Key: Owner
Value: !Ref Owner
- Key: Project
Value: !Ref Project
FunctionName: "Batch-Add-Jobs-To-Queue-New"
Handler: AddJobToQueue.handler
Role: !GetAtt BatchLambdaRole.Arn
Runtime: nodejs14.x

BatchLambdaRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service: "lambda.amazonaws.com"
Action:
- 'sts:AssumeRole'
Policies:
- PolicyName: "dev-batch-lambda-policy"
PolicyDocument:
Statement:
- Effect: "Allow"
Resource: "*"
Action:
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
- 'batch:DescribeJobs'
- 'batch:SubmitJob'
- 'batch:ListJobs'

Tags:
- Key: Name
Value: !Join [ '-', [ !Ref EnvironmentName, !Ref ApplicationName, 'batchLambdaRole' ] ]
- Key: Environment
Value: !Ref EnvironmentName
- Key: Owner
Value: !Ref Owner
- Key: Project
Value: !Ref Project

Outputs:
BatchLambdaOutput:
Description: Arn of lambda for adding jobs to queue
Value: !GetAtt BatchLambda.Arn
Export:
Name: !Sub "BatchLambdaID"
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
[
{
"ParameterKey": "DevBucketName",
"ParameterValue": "dev-batch-lambda-deployment-bucket"
},
{
"ParameterKey": "ProdBucketName",
"ParameterValue": "prod-batch-lambda-deployment-bucket"
},
{
"ParameterKey": "LambdaCodeS3Key",
"ParameterValue": "deployment_package.zip"
},
{
"ParameterKey": "CodeCommitRepositoryName",
"ParameterValue": "Batch-Lambda"
},
{
"ParameterKey": "CodeCommitBranch",
"ParameterValue": "master"
}
]
Loading