Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Implement write access sharing #377

Merged
merged 54 commits into from
Jun 27, 2024
Merged
Show file tree
Hide file tree
Changes from 46 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
5e51ea2
Initial implementation.
Oleksii-Klimov Jun 20, 2024
ce9a1e9
Update tests.
Oleksii-Klimov Jun 20, 2024
9365bf6
Fix formatting.
Oleksii-Klimov Jun 20, 2024
e24c602
Fix flaky tests.
Oleksii-Klimov Jun 20, 2024
dfc1c77
Fix more tests.
Oleksii-Klimov Jun 20, 2024
c64e330
Missing changes.
Oleksii-Klimov Jun 20, 2024
1704166
Revert some changes.
Oleksii-Klimov Jun 20, 2024
45fe02a
Add write sharing test.
Oleksii-Klimov Jun 20, 2024
20425f9
Implement revoke permissions.
Oleksii-Klimov Jun 20, 2024
263aec5
Address review comments.
Oleksii-Klimov Jun 21, 2024
b0a950d
Revoke all if no specific permissions.
Oleksii-Klimov Jun 21, 2024
0c065a6
Minor fixes.
Oleksii-Klimov Jun 21, 2024
2535289
Revert some changes.
Oleksii-Klimov Jun 21, 2024
3d7ab6c
Change set to list where makes sense.
Oleksii-Klimov Jun 21, 2024
4b7753c
Make list modifiable.
Oleksii-Klimov Jun 21, 2024
0526f78
Optimize shared permissions check.
Oleksii-Klimov Jun 24, 2024
2e9f073
Fix recursive map update.
Oleksii-Klimov Jun 24, 2024
71e612a
Small optimization.
Oleksii-Klimov Jun 24, 2024
9fe3517
NPE fix.
Oleksii-Klimov Jun 24, 2024
57ee022
Fix condition.
Oleksii-Klimov Jun 24, 2024
93d8f0a
Can't create enum set from an empty set.
Oleksii-Klimov Jun 24, 2024
83d05be
Minor fixes.
Oleksii-Klimov Jun 24, 2024
9113378
Remove unused imports.
Oleksii-Klimov Jun 24, 2024
e4b0020
Correct defaults.
Oleksii-Klimov Jun 24, 2024
ac6a758
Address review comments.
Oleksii-Klimov Jun 25, 2024
509c56e
Missing optimization.
Oleksii-Klimov Jun 25, 2024
a7fc573
Fix formatting.
Oleksii-Klimov Jun 25, 2024
8d64a06
Fix typo.
Oleksii-Klimov Jun 25, 2024
3e7941a
Remove extra asterisk
Oleksii-Klimov Jun 25, 2024
965ab4c
Fix publication test.
Oleksii-Klimov Jun 25, 2024
d2dfcd4
Fix invitation removal.
Oleksii-Klimov Jun 25, 2024
941c515
Refactor cleanup and revoke methods for a all permissions.
Oleksii-Klimov Jun 25, 2024
23b4613
Make permissions mutable.
Oleksii-Klimov Jun 26, 2024
e0252fb
Minor update.
Oleksii-Klimov Jun 26, 2024
df084c6
Clarify names.
Oleksii-Klimov Jun 26, 2024
1385861
Fix invitations cleanup.
Oleksii-Klimov Jun 26, 2024
a5abf82
Refactor SharedByMeDto.
Oleksii-Klimov Jun 26, 2024
e53c670
Fix npe.
Oleksii-Klimov Jun 26, 2024
75026b7
Fix another npe.
Oleksii-Klimov Jun 26, 2024
cbdb29e
Remove permission fetcher.
Oleksii-Klimov Jun 26, 2024
8d57f37
Update tests.
Oleksii-Klimov Jun 26, 2024
f06ddfd
Simplify code.
Oleksii-Klimov Jun 26, 2024
31ce59f
Filter public resources inside rule service.
Oleksii-Klimov Jun 26, 2024
18c9a01
Fix npe.
Oleksii-Klimov Jun 26, 2024
1f644a5
Remove redundant test.
Oleksii-Klimov Jun 27, 2024
d50fff4
Remove redundant check.
Oleksii-Klimov Jun 27, 2024
365c009
Remove redundant line.
Oleksii-Klimov Jun 27, 2024
720bc11
Address review comments.
Oleksii-Klimov Jun 27, 2024
04b2c1a
Merge branch 'development' into 376-implement-write-access-sharing
Oleksii-Klimov Jun 27, 2024
6995705
Fix typos.
Oleksii-Klimov Jun 27, 2024
ac4c279
Add a test for partial invitation cleanup.
Oleksii-Klimov Jun 27, 2024
9572eba
Check if sorting is still needed.
Oleksii-Klimov Jun 27, 2024
9fca1c7
Change enum erder.
Oleksii-Klimov Jun 27, 2024
8bc7a14
Ignore unknown properties for forward compatibility.
Oleksii-Klimov Jun 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions src/main/java/com/epam/aidial/core/ProxyContext.java
Original file line number Diff line number Diff line change
Expand Up @@ -134,4 +134,8 @@ public String getProject() {
public List<String> getExecutionPath() {
return proxyApiKeyData == null ? null : proxyApiKeyData.getExecutionPath();
}

public boolean getBooleanParam(String name) {
Maxim-Gadalov marked this conversation as resolved.
Show resolved Hide resolved
return Boolean.parseBoolean(request.getParam(name, "false"));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import com.epam.aidial.core.Proxy;
import com.epam.aidial.core.ProxyContext;
import com.epam.aidial.core.data.ResourceAccessType;
import com.epam.aidial.core.security.AccessService;
import com.epam.aidial.core.storage.ResourceDescription;
import com.epam.aidial.core.util.HttpStatus;
Expand Down Expand Up @@ -29,7 +30,8 @@ public Future<?> handle(String resourceUrl) {
return proxy.getVertx()
.executeBlocking(() -> {
AccessService service = proxy.getAccessService();
return isWriteAccess ? service.hasWriteAccess(resource, context) : service.hasReadAccess(resource, context);
ResourceAccessType permission = isWriteAccess ? ResourceAccessType.WRITE : ResourceAccessType.READ;
return service.hasAccess(resource, context, permission);
}, false)
.map(hasAccess -> {
if (hasAccess) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@

import com.epam.aidial.core.Proxy;
import com.epam.aidial.core.ProxyContext;
import com.epam.aidial.core.data.ResourceLink;
import com.epam.aidial.core.data.ResourceLinkCollection;
import com.epam.aidial.core.service.InvitationService;
import com.epam.aidial.core.service.LockService;
import com.epam.aidial.core.service.ShareService;
Expand All @@ -13,9 +11,6 @@
import io.vertx.core.Future;
import lombok.extern.slf4j.Slf4j;

import java.util.HashSet;
import java.util.Set;

@Slf4j
public class DeleteFileController extends AccessControlBaseController {

Expand Down Expand Up @@ -43,11 +38,9 @@ protected Future<?> handle(ResourceDescription resource) {
String bucketName = resource.getBucketName();
String bucketLocation = resource.getBucketLocation();
try {
Set<ResourceLink> resourceLinks = new HashSet<>();
resourceLinks.add(new ResourceLink(resource.getUrl()));
return lockService.underBucketLock(bucketLocation, () -> {
invitationService.cleanUpResourceLinks(bucketName, bucketLocation, resourceLinks);
shareService.revokeSharedAccess(bucketName, bucketLocation, new ResourceLinkCollection(resourceLinks));
invitationService.cleanUpResourceLink(bucketName, bucketLocation, resource);
shareService.revokeSharedResource(bucketName, bucketLocation, resource);
storage.delete(absoluteFilePath);
return null;
});
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,23 @@
import com.epam.aidial.core.Proxy;
import com.epam.aidial.core.ProxyContext;
import com.epam.aidial.core.data.MetadataBase;
import com.epam.aidial.core.security.AccessService;
import com.epam.aidial.core.storage.BlobStorage;
import com.epam.aidial.core.storage.ResourceDescription;
import com.epam.aidial.core.util.HttpStatus;
import io.vertx.core.Future;
import io.vertx.core.http.HttpHeaders;
import lombok.extern.slf4j.Slf4j;

import java.util.List;

@Slf4j
public class FileMetadataController extends AccessControlBaseController {
private final AccessService accessService;

public FileMetadataController(Proxy proxy, ProxyContext context) {
super(proxy, context, false);
accessService = proxy.getAccessService();
}

private String getContentType() {
Expand All @@ -37,7 +42,10 @@ protected Future<?> handle(ResourceDescription resource) {
try {
MetadataBase metadata = storage.listMetadata(resource, token, limit, recursive);
if (metadata != null) {
proxy.getAccessService().filterForbidden(context, resource, metadata);
accessService.filterForbidden(context, resource, metadata);
if (context.getBooleanParam("permissions")) {
accessService.populatePermissions(context, resource.getBucketLocation(), List.of(metadata));
}
context.respond(HttpStatus.OK, getContentType(), metadata);
} else {
context.respond(HttpStatus.NOT_FOUND);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import com.epam.aidial.core.Proxy;
import com.epam.aidial.core.ProxyContext;
import com.epam.aidial.core.data.ListPublishedResourcesRequest;
import com.epam.aidial.core.data.MetadataBase;
import com.epam.aidial.core.data.Publication;
import com.epam.aidial.core.data.Publications;
import com.epam.aidial.core.data.RejectPublicationRequest;
Expand All @@ -26,6 +27,9 @@
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;

import java.util.Collection;
import java.util.List;

@Slf4j
@RequiredArgsConstructor
public class PublicationController {
Expand Down Expand Up @@ -161,7 +165,14 @@ public Future<?> listPublishedResources() {
ListPublishedResourcesRequest request = ProxyUtil.convertToObject(body, ListPublishedResourcesRequest.class);
String bucketLocation = BlobStorageUtil.buildInitiatorBucket(context);
String bucket = encryptService.encrypt(bucketLocation);
return vertx.executeBlocking(() -> publicationService.listPublishedResources(request, bucket, bucketLocation), false);
return vertx.executeBlocking(() -> {
Collection<MetadataBase> metadata =
publicationService.listPublishedResources(request, bucket, bucketLocation);
if (context.getBooleanParam("permissions")) {
accessService.populatePermissions(context, bucketLocation, metadata);
}
return metadata;
}, false);
})
.onSuccess(metadata -> context.respond(HttpStatus.OK, metadata))
.onFailure(error -> respondError("Can't list published resources", error));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,8 @@
import com.epam.aidial.core.data.Conversation;
import com.epam.aidial.core.data.MetadataBase;
import com.epam.aidial.core.data.Prompt;
import com.epam.aidial.core.data.ResourceLink;
import com.epam.aidial.core.data.ResourceLinkCollection;
import com.epam.aidial.core.data.ResourceType;
import com.epam.aidial.core.security.AccessService;
import com.epam.aidial.core.service.InvitationService;
import com.epam.aidial.core.service.LockService;
import com.epam.aidial.core.service.ResourceService;
Expand All @@ -23,8 +22,7 @@
import lombok.extern.slf4j.Slf4j;

import java.nio.charset.StandardCharsets;
import java.util.HashSet;
import java.util.Set;
import java.util.List;

@Slf4j
@SuppressWarnings("checkstyle:Indentation")
Expand All @@ -36,13 +34,15 @@ public class ResourceController extends AccessControlBaseController {
private final LockService lockService;
private final InvitationService invitationService;
private final boolean metadata;
private final AccessService accessService;

public ResourceController(Proxy proxy, ProxyContext context, boolean metadata) {
// PUT and DELETE require full access, GET - not
// PUT and DELETE require write access, GET - read
super(proxy, context, !HttpMethod.GET.equals(context.getRequest().method()));
this.vertx = proxy.getVertx();
this.service = proxy.getResourceService();
this.shareService = proxy.getShareService();
this.accessService = proxy.getAccessService();
this.lockService = proxy.getLockService();
this.invitationService = proxy.getInvitationService();
this.metadata = metadata;
Expand Down Expand Up @@ -93,7 +93,10 @@ private Future<?> getMetadata(ResourceDescription descriptor) {
if (result == null) {
context.respond(HttpStatus.NOT_FOUND, "Not found: " + descriptor.getUrl());
} else {
proxy.getAccessService().filterForbidden(context, descriptor, result);
accessService.filterForbidden(context, descriptor, result);
if (context.getBooleanParam("permissions")) {
accessService.populatePermissions(context, descriptor.getBucketLocation(), List.of(result));
}
context.respond(HttpStatus.OK, getContentType(), result);
}
})
Expand Down Expand Up @@ -188,14 +191,11 @@ private Future<?> deleteResource(ResourceDescription descriptor) {
}

return vertx.executeBlocking(() -> {
Set<ResourceLink> resourceLinks = new HashSet<>();
resourceLinks.add(new ResourceLink(descriptor.getUrl()));
String bucketName = descriptor.getBucketName();
String bucketLocation = descriptor.getBucketLocation();
return lockService.underBucketLock(bucketLocation, () -> {
invitationService.cleanUpResourceLinks(bucketName, bucketLocation, resourceLinks);
shareService.revokeSharedAccess(bucketName, bucketLocation,
new ResourceLinkCollection(resourceLinks));
invitationService.cleanUpResourceLink(bucketName, bucketLocation, descriptor);
shareService.revokeSharedResource(bucketName, bucketLocation, descriptor);
return service.deleteResource(descriptor);
});
}, false)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,25 +4,29 @@
import com.epam.aidial.core.ProxyContext;
import com.epam.aidial.core.data.CopySharedAccessRequest;
import com.epam.aidial.core.data.ListSharedResourcesRequest;
import com.epam.aidial.core.data.ResourceAccessType;
import com.epam.aidial.core.data.ResourceLinkCollection;
import com.epam.aidial.core.data.RevokeResourcesRequest;
import com.epam.aidial.core.data.ShareResourcesRequest;
import com.epam.aidial.core.data.SharedResource;
import com.epam.aidial.core.security.EncryptionService;
import com.epam.aidial.core.service.InvitationService;
import com.epam.aidial.core.service.LockService;
import com.epam.aidial.core.service.ResourceService;
import com.epam.aidial.core.service.ShareService;
import com.epam.aidial.core.storage.BlobStorage;
import com.epam.aidial.core.storage.BlobStorageUtil;
import com.epam.aidial.core.storage.ResourceDescription;
import com.epam.aidial.core.util.HttpException;
import com.epam.aidial.core.util.HttpStatus;
import com.epam.aidial.core.util.ProxyUtil;
import com.epam.aidial.core.util.ResourceUtil;
import com.google.common.collect.Sets;
import io.vertx.core.Future;
import io.vertx.core.buffer.Buffer;
import lombok.extern.slf4j.Slf4j;

import java.nio.charset.StandardCharsets;
import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;

@Slf4j
public class ShareController {
Expand All @@ -35,8 +39,6 @@ public class ShareController {
private final EncryptionService encryptionService;
private final LockService lockService;
private final InvitationService invitationService;
private final ResourceService resourceService;
private final BlobStorage storage;

public ShareController(Proxy proxy, ProxyContext context) {
this.proxy = proxy;
Expand All @@ -45,8 +47,6 @@ public ShareController(Proxy proxy, ProxyContext context) {
this.encryptionService = proxy.getEncryptionService();
this.lockService = proxy.getLockService();
this.invitationService = proxy.getInvitationService();
this.resourceService = proxy.getResourceService();
this.storage = proxy.getStorage();
}

public Future<?> handle(Operation operation) {
Expand Down Expand Up @@ -133,13 +133,18 @@ public Future<?> revokeSharedResources() {
return context.getRequest()
.body()
.compose(buffer -> {
ResourceLinkCollection request = getResourceLinkCollection(buffer, Operation.REVOKE);
RevokeResourcesRequest request = getRevokeResourcesRequest(buffer, Operation.REVOKE);
Maxim-Gadalov marked this conversation as resolved.
Show resolved Hide resolved
String bucketLocation = BlobStorageUtil.buildInitiatorBucket(context);
String bucket = encryptionService.encrypt(bucketLocation);
Map<ResourceDescription, Set<ResourceAccessType>> permissionsToRevoke = request.getResources().stream()
.collect(Collectors.toUnmodifiableMap(
resource -> shareService.getResourceFromLink(resource.url()),
SharedResource::permissions,
Sets::union));
return proxy.getVertx()
.executeBlocking(() -> lockService.underBucketLock(bucketLocation, () -> {
invitationService.cleanUpResourceLinks(bucket, bucketLocation, request.getResources());
shareService.revokeSharedAccess(bucket, bucketLocation, request);
invitationService.cleanUpPermissions(bucket, bucketLocation, permissionsToRevoke);
shareService.revokeSharedAccess(bucket, bucketLocation, permissionsToRevoke);
return null;
}), false);
})
Expand Down Expand Up @@ -210,8 +215,14 @@ private ResourceLinkCollection getResourceLinkCollection(Buffer buffer, Operatio
}
}

private boolean hasResource(ResourceDescription resource) {
return ResourceUtil.hasResource(resource, resourceService, storage);
private RevokeResourcesRequest getRevokeResourcesRequest(Buffer buffer, Operation operation) {
try {
String body = buffer.toString(StandardCharsets.UTF_8);
return ProxyUtil.convertToObject(body, RevokeResourcesRequest.class);
} catch (Exception e) {
log.error("Invalid request body provided", e);
throw new HttpException(HttpStatus.BAD_REQUEST, "Can't %s shared resources. Incorrect body".formatted(operation));
}
}

public enum Operation {
Expand Down
23 changes: 19 additions & 4 deletions src/main/java/com/epam/aidial/core/data/Invitation.java
Original file line number Diff line number Diff line change
@@ -1,17 +1,32 @@
package com.epam.aidial.core.data;

import lombok.AllArgsConstructor;
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.Data;
import lombok.NoArgsConstructor;

import java.util.Set;
import java.util.List;
import java.util.stream.Collectors;

@Data
@AllArgsConstructor
@NoArgsConstructor
public class Invitation {
String id;
Set<ResourceLink> resources;
List<SharedResource> resources;
long createdAt;
long expireAt;

@JsonCreator
public Invitation(
@JsonProperty("id") String id,
@JsonProperty("resources") List<SharedResource> resources,
@JsonProperty("createdAt") long createdAt,
@JsonProperty("expireAt") long expireAt) {
this.id = id;
this.resources = resources.stream()
.map(SharedResource::withReadIfNoPermissions)
.collect(Collectors.toList());
this.createdAt = createdAt;
this.expireAt = expireAt;
}
}
5 changes: 5 additions & 0 deletions src/main/java/com/epam/aidial/core/data/MetadataBase.java
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
package com.epam.aidial.core.data;

import com.fasterxml.jackson.annotation.JsonInclude;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

import java.util.Set;

@AllArgsConstructor
@Data
@NoArgsConstructor
Expand All @@ -16,4 +19,6 @@ public abstract class MetadataBase {
private String url;
private NodeType nodeType;
private ResourceType resourceType;
@JsonInclude(JsonInclude.Include.NON_NULL)
private Set<ResourceAccessType> permissions;
}
15 changes: 15 additions & 0 deletions src/main/java/com/epam/aidial/core/data/ResourceAccessType.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
package com.epam.aidial.core.data;

import java.util.Collections;
import java.util.EnumSet;
import java.util.Set;

public enum ResourceAccessType {
READ,
WRITE;

public static final Set<ResourceAccessType> ALL = Collections.unmodifiableSet(
EnumSet.allOf(ResourceAccessType.class));
public static final Set<ResourceAccessType> READ_ONLY = Collections.unmodifiableSet(
EnumSet.of(READ));
}
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ public class ResourceFolderMetadata extends MetadataBase {
private String nextToken;

public ResourceFolderMetadata(ResourceType type, String bucket, String name, String path, String url, List<MetadataBase> items) {
super(name, path, bucket, url, NodeType.FOLDER, type);
super(name, path, bucket, url, NodeType.FOLDER, type, null);
this.items = items;
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ public class ResourceItemMetadata extends MetadataBase {
private Long updatedAt;

public ResourceItemMetadata(ResourceType type, String bucket, String name, String path, String url) {
super(name, path, bucket, url, NodeType.ITEM, type);
super(name, path, bucket, url, NodeType.ITEM, type, null);
}

public ResourceItemMetadata(ResourceDescription resource) {
Expand Down
Loading
Loading