Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Implement write access sharing #377

Merged
merged 54 commits into from
Jun 27, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
5e51ea2
Initial implementation.
Oleksii-Klimov Jun 20, 2024
ce9a1e9
Update tests.
Oleksii-Klimov Jun 20, 2024
9365bf6
Fix formatting.
Oleksii-Klimov Jun 20, 2024
e24c602
Fix flaky tests.
Oleksii-Klimov Jun 20, 2024
dfc1c77
Fix more tests.
Oleksii-Klimov Jun 20, 2024
c64e330
Missing changes.
Oleksii-Klimov Jun 20, 2024
1704166
Revert some changes.
Oleksii-Klimov Jun 20, 2024
45fe02a
Add write sharing test.
Oleksii-Klimov Jun 20, 2024
20425f9
Implement revoke permissions.
Oleksii-Klimov Jun 20, 2024
263aec5
Address review comments.
Oleksii-Klimov Jun 21, 2024
b0a950d
Revoke all if no specific permissions.
Oleksii-Klimov Jun 21, 2024
0c065a6
Minor fixes.
Oleksii-Klimov Jun 21, 2024
2535289
Revert some changes.
Oleksii-Klimov Jun 21, 2024
3d7ab6c
Change set to list where makes sense.
Oleksii-Klimov Jun 21, 2024
4b7753c
Make list modifiable.
Oleksii-Klimov Jun 21, 2024
0526f78
Optimize shared permissions check.
Oleksii-Klimov Jun 24, 2024
2e9f073
Fix recursive map update.
Oleksii-Klimov Jun 24, 2024
71e612a
Small optimization.
Oleksii-Klimov Jun 24, 2024
9fe3517
NPE fix.
Oleksii-Klimov Jun 24, 2024
57ee022
Fix condition.
Oleksii-Klimov Jun 24, 2024
93d8f0a
Can't create enum set from an empty set.
Oleksii-Klimov Jun 24, 2024
83d05be
Minor fixes.
Oleksii-Klimov Jun 24, 2024
9113378
Remove unused imports.
Oleksii-Klimov Jun 24, 2024
e4b0020
Correct defaults.
Oleksii-Klimov Jun 24, 2024
ac6a758
Address review comments.
Oleksii-Klimov Jun 25, 2024
509c56e
Missing optimization.
Oleksii-Klimov Jun 25, 2024
a7fc573
Fix formatting.
Oleksii-Klimov Jun 25, 2024
8d64a06
Fix typo.
Oleksii-Klimov Jun 25, 2024
3e7941a
Remove extra asterisk
Oleksii-Klimov Jun 25, 2024
965ab4c
Fix publication test.
Oleksii-Klimov Jun 25, 2024
d2dfcd4
Fix invitation removal.
Oleksii-Klimov Jun 25, 2024
941c515
Refactor cleanup and revoke methods for a all permissions.
Oleksii-Klimov Jun 25, 2024
23b4613
Make permissions mutable.
Oleksii-Klimov Jun 26, 2024
e0252fb
Minor update.
Oleksii-Klimov Jun 26, 2024
df084c6
Clarify names.
Oleksii-Klimov Jun 26, 2024
1385861
Fix invitations cleanup.
Oleksii-Klimov Jun 26, 2024
a5abf82
Refactor SharedByMeDto.
Oleksii-Klimov Jun 26, 2024
e53c670
Fix npe.
Oleksii-Klimov Jun 26, 2024
75026b7
Fix another npe.
Oleksii-Klimov Jun 26, 2024
cbdb29e
Remove permission fetcher.
Oleksii-Klimov Jun 26, 2024
8d57f37
Update tests.
Oleksii-Klimov Jun 26, 2024
f06ddfd
Simplify code.
Oleksii-Klimov Jun 26, 2024
31ce59f
Filter public resources inside rule service.
Oleksii-Klimov Jun 26, 2024
18c9a01
Fix npe.
Oleksii-Klimov Jun 26, 2024
1f644a5
Remove redundant test.
Oleksii-Klimov Jun 27, 2024
d50fff4
Remove redundant check.
Oleksii-Klimov Jun 27, 2024
365c009
Remove redundant line.
Oleksii-Klimov Jun 27, 2024
720bc11
Address review comments.
Oleksii-Klimov Jun 27, 2024
04b2c1a
Merge branch 'development' into 376-implement-write-access-sharing
Oleksii-Klimov Jun 27, 2024
6995705
Fix typos.
Oleksii-Klimov Jun 27, 2024
ac4c279
Add a test for partial invitation cleanup.
Oleksii-Klimov Jun 27, 2024
9572eba
Check if sorting is still needed.
Oleksii-Klimov Jun 27, 2024
9fca1c7
Change enum erder.
Oleksii-Klimov Jun 27, 2024
8bc7a14
Ignore unknown properties for forward compatibility.
Oleksii-Klimov Jun 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,22 @@

import com.epam.aidial.core.Proxy;
import com.epam.aidial.core.ProxyContext;
import com.epam.aidial.core.data.ResourceAccessType;
import com.epam.aidial.core.security.AccessService;
import com.epam.aidial.core.storage.ResourceDescription;
import com.epam.aidial.core.util.HttpStatus;
import io.vertx.core.Future;
import lombok.AllArgsConstructor;

import java.util.List;
import java.util.Set;

@AllArgsConstructor
public abstract class AccessControlBaseController {

final Proxy proxy;
final ProxyContext context;
final boolean isWriteAccess;
final ResourceAccessType accessType;
Maxim-Gadalov marked this conversation as resolved.
Show resolved Hide resolved

public Future<?> handle(String resourceUrl) {
ResourceDescription resource;
Expand All @@ -29,7 +33,7 @@ public Future<?> handle(String resourceUrl) {
return proxy.getVertx()
.executeBlocking(() -> {
AccessService service = proxy.getAccessService();
return isWriteAccess ? service.hasWriteAccess(resource, context) : service.hasReadAccess(resource, context);
return !service.lookupPermissions(resource, context, Set.of(accessType)).isEmpty();
}, false)
.map(hasAccess -> {
if (hasAccess) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import com.epam.aidial.core.Proxy;
import com.epam.aidial.core.ProxyContext;
import com.epam.aidial.core.data.ResourceAccessType;
import com.epam.aidial.core.data.ResourceLink;
import com.epam.aidial.core.data.ResourceLinkCollection;
import com.epam.aidial.core.service.InvitationService;
Expand All @@ -24,7 +25,7 @@ public class DeleteFileController extends AccessControlBaseController {
private final LockService lockService;

public DeleteFileController(Proxy proxy, ProxyContext context) {
super(proxy, context, true);
super(proxy, context, ResourceAccessType.WRITE);
this.shareService = proxy.getShareService();
this.invitationService = proxy.getInvitationService();
this.lockService = proxy.getLockService();
Expand All @@ -47,7 +48,7 @@ protected Future<?> handle(ResourceDescription resource) {
resourceLinks.add(new ResourceLink(resource.getUrl()));
return lockService.underBucketLock(bucketLocation, () -> {
invitationService.cleanUpResourceLinks(bucketName, bucketLocation, resourceLinks);
shareService.revokeSharedAccess(bucketName, bucketLocation, new ResourceLinkCollection(resourceLinks));
shareService.revokeSharedAccess(bucketName, bucketLocation, resourceLinks);
storage.delete(absoluteFilePath);
return null;
});
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import com.epam.aidial.core.Proxy;
import com.epam.aidial.core.ProxyContext;
import com.epam.aidial.core.data.ResourceAccessType;
import com.epam.aidial.core.storage.InputStreamReader;
import com.epam.aidial.core.storage.ResourceDescription;
import com.epam.aidial.core.util.HttpStatus;
Expand All @@ -20,7 +21,7 @@
public class DownloadFileController extends AccessControlBaseController {

public DownloadFileController(Proxy proxy, ProxyContext context) {
super(proxy, context, false);
super(proxy, context, ResourceAccessType.READ);
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
import com.epam.aidial.core.Proxy;
import com.epam.aidial.core.ProxyContext;
import com.epam.aidial.core.data.MetadataBase;
import com.epam.aidial.core.data.ResourceAccessType;
import com.epam.aidial.core.security.AccessService;
import com.epam.aidial.core.service.PermissionsFetcher;
import com.epam.aidial.core.storage.BlobStorage;
import com.epam.aidial.core.storage.ResourceDescription;
import com.epam.aidial.core.util.HttpStatus;
Expand All @@ -12,9 +15,11 @@

@Slf4j
public class FileMetadataController extends AccessControlBaseController {
private final AccessService accessService;

public FileMetadataController(Proxy proxy, ProxyContext context) {
super(proxy, context, false);
super(proxy, context, ResourceAccessType.READ);
accessService = proxy.getAccessService();
}

private String getContentType() {
Expand All @@ -33,9 +38,10 @@ protected Future<?> handle(ResourceDescription resource) {
if (limit < 0 || limit > 1000) {
return context.respond(HttpStatus.BAD_REQUEST, "Limit is out of allowed range: [0, 1000]");
}
PermissionsFetcher permissionsFetcher = PermissionsFetcher.of(context, accessService);
return proxy.getVertx().executeBlocking(() -> {
try {
MetadataBase metadata = storage.listMetadata(resource, token, limit, recursive);
MetadataBase metadata = storage.listMetadata(resource, permissionsFetcher, token, limit, recursive);
Maxim-Gadalov marked this conversation as resolved.
Show resolved Hide resolved
if (metadata != null) {
proxy.getAccessService().filterForbidden(context, resource, metadata);
context.respond(HttpStatus.OK, getContentType(), metadata);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
import com.epam.aidial.core.security.EncryptionService;
import com.epam.aidial.core.service.LockService;
import com.epam.aidial.core.service.PermissionDeniedException;
import com.epam.aidial.core.service.PermissionsFetcher;
import com.epam.aidial.core.service.PublicationService;
import com.epam.aidial.core.service.ResourceNotFoundException;
import com.epam.aidial.core.service.RuleService;
Expand Down Expand Up @@ -161,7 +162,9 @@ public Future<?> listPublishedResources() {
ListPublishedResourcesRequest request = ProxyUtil.convertToObject(body, ListPublishedResourcesRequest.class);
String bucketLocation = BlobStorageUtil.buildInitiatorBucket(context);
String bucket = encryptService.encrypt(bucketLocation);
return vertx.executeBlocking(() -> publicationService.listPublishedResources(request, bucket, bucketLocation), false);
PermissionsFetcher permissionsFetcher = PermissionsFetcher.of(context, accessService);
return vertx.executeBlocking(() -> publicationService.listPublishedResources(
Maxim-Gadalov marked this conversation as resolved.
Show resolved Hide resolved
request, permissionsFetcher, bucket, bucketLocation), false);
})
.onSuccess(metadata -> context.respond(HttpStatus.OK, metadata))
.onFailure(error -> respondError("Can't list published resources", error));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,13 @@
import com.epam.aidial.core.data.Conversation;
import com.epam.aidial.core.data.MetadataBase;
import com.epam.aidial.core.data.Prompt;
import com.epam.aidial.core.data.ResourceAccessType;
import com.epam.aidial.core.data.ResourceLink;
import com.epam.aidial.core.data.ResourceLinkCollection;
import com.epam.aidial.core.data.ResourceType;
import com.epam.aidial.core.security.AccessService;
import com.epam.aidial.core.service.InvitationService;
import com.epam.aidial.core.service.LockService;
import com.epam.aidial.core.service.PermissionsFetcher;
import com.epam.aidial.core.service.ResourceService;
import com.epam.aidial.core.service.ShareService;
import com.epam.aidial.core.storage.ResourceDescription;
Expand All @@ -36,13 +38,19 @@ public class ResourceController extends AccessControlBaseController {
private final LockService lockService;
private final InvitationService invitationService;
private final boolean metadata;
private final AccessService accessService;

public ResourceController(Proxy proxy, ProxyContext context, boolean metadata) {
// PUT and DELETE require full access, GET - not
super(proxy, context, !HttpMethod.GET.equals(context.getRequest().method()));
// PUT and DELETE require write access, GET - read
super(
proxy,
context,
HttpMethod.GET.equals(context.getRequest().method())
? ResourceAccessType.READ : ResourceAccessType.WRITE);
this.vertx = proxy.getVertx();
this.service = proxy.getResourceService();
this.shareService = proxy.getShareService();
this.accessService = proxy.getAccessService();
this.lockService = proxy.getLockService();
this.invitationService = proxy.getInvitationService();
this.metadata = metadata;
Expand Down Expand Up @@ -88,7 +96,9 @@ private Future<?> getMetadata(ResourceDescription descriptor) {
return context.respond(HttpStatus.BAD_REQUEST, "Bad query parameters. Limit must be in [0, 1000] range. Recursive must be true/false");
}

return vertx.executeBlocking(() -> service.getMetadata(descriptor, token, limit, recursive), false)
PermissionsFetcher permissionsFetcher = PermissionsFetcher.of(context, accessService);
return vertx.executeBlocking(() -> service.getMetadata(
descriptor, permissionsFetcher, token, limit, recursive), false)
.onSuccess(result -> {
if (result == null) {
context.respond(HttpStatus.NOT_FOUND, "Not found: " + descriptor.getUrl());
Expand Down Expand Up @@ -194,8 +204,7 @@ private Future<?> deleteResource(ResourceDescription descriptor) {
String bucketLocation = descriptor.getBucketLocation();
return lockService.underBucketLock(bucketLocation, () -> {
invitationService.cleanUpResourceLinks(bucketName, bucketLocation, resourceLinks);
shareService.revokeSharedAccess(bucketName, bucketLocation,
new ResourceLinkCollection(resourceLinks));
shareService.revokeSharedAccess(bucketName, bucketLocation, resourceLinks);
return service.deleteResource(descriptor);
});
}, false)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import com.epam.aidial.core.data.CopySharedAccessRequest;
import com.epam.aidial.core.data.ListSharedResourcesRequest;
import com.epam.aidial.core.data.ResourceLinkCollection;
import com.epam.aidial.core.data.RevokeResourcesRequest;
import com.epam.aidial.core.data.ShareResourcesRequest;
import com.epam.aidial.core.security.EncryptionService;
import com.epam.aidial.core.service.InvitationService;
Expand Down Expand Up @@ -35,8 +36,6 @@ public class ShareController {
private final EncryptionService encryptionService;
private final LockService lockService;
private final InvitationService invitationService;
private final ResourceService resourceService;
private final BlobStorage storage;

public ShareController(Proxy proxy, ProxyContext context) {
this.proxy = proxy;
Expand All @@ -45,8 +44,6 @@ public ShareController(Proxy proxy, ProxyContext context) {
this.encryptionService = proxy.getEncryptionService();
this.lockService = proxy.getLockService();
this.invitationService = proxy.getInvitationService();
this.resourceService = proxy.getResourceService();
this.storage = proxy.getStorage();
}

public Future<?> handle(Operation operation) {
Expand Down Expand Up @@ -133,13 +130,13 @@ public Future<?> revokeSharedResources() {
return context.getRequest()
.body()
.compose(buffer -> {
ResourceLinkCollection request = getResourceLinkCollection(buffer, Operation.REVOKE);
RevokeResourcesRequest request = getRevokeResourcesRequest(buffer, Operation.REVOKE);
Maxim-Gadalov marked this conversation as resolved.
Show resolved Hide resolved
String bucketLocation = BlobStorageUtil.buildInitiatorBucket(context);
String bucket = encryptionService.encrypt(bucketLocation);
return proxy.getVertx()
.executeBlocking(() -> lockService.underBucketLock(bucketLocation, () -> {
invitationService.cleanUpResourceLinks(bucket, bucketLocation, request.getResources());
shareService.revokeSharedAccess(bucket, bucketLocation, request);
shareService.revokeSharedAccess(bucket, bucketLocation, request.getResources());
return null;
}), false);
})
Expand Down Expand Up @@ -210,8 +207,14 @@ private ResourceLinkCollection getResourceLinkCollection(Buffer buffer, Operatio
}
}

private boolean hasResource(ResourceDescription resource) {
return ResourceUtil.hasResource(resource, resourceService, storage);
private RevokeResourcesRequest getRevokeResourcesRequest(Buffer buffer, Operation operation) {
try {
String body = buffer.toString(StandardCharsets.UTF_8);
return ProxyUtil.convertToObject(body, RevokeResourcesRequest.class);
} catch (Exception e) {
log.error("Invalid request body provided", e);
throw new HttpException(HttpStatus.BAD_REQUEST, "Can't %s shared resources. Incorrect body".formatted(operation));
}
}

public enum Operation {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import com.epam.aidial.core.Proxy;
import com.epam.aidial.core.ProxyContext;
import com.epam.aidial.core.data.ResourceAccessType;
import com.epam.aidial.core.storage.BlobWriteStream;
import com.epam.aidial.core.storage.ResourceDescription;
import com.epam.aidial.core.util.HttpStatus;
Expand All @@ -16,7 +17,7 @@
public class UploadFileController extends AccessControlBaseController {

public UploadFileController(Proxy proxy, ProxyContext context) {
super(proxy, context, true);
super(proxy, context, ResourceAccessType.WRITE);
}

@Override
Expand Down
18 changes: 14 additions & 4 deletions src/main/java/com/epam/aidial/core/data/FileMetadata.java
Original file line number Diff line number Diff line change
Expand Up @@ -5,21 +5,31 @@
import lombok.NoArgsConstructor;
import lombok.Setter;

import java.util.Set;

@Getter
@Setter
@NoArgsConstructor
public class FileMetadata extends ResourceItemMetadata {
long contentLength;
String contentType;

public FileMetadata(String bucket, String name, String path, String url, long contentLength, String contentType) {
super(ResourceType.FILE, bucket, name, path, url);
public FileMetadata(
String bucket,
String name,
String path,
String url,
long contentLength,
String contentType,
Set<ResourceAccessType> permissions) {
super(ResourceType.FILE, bucket, name, path, url, permissions);
this.contentLength = contentLength;
this.contentType = contentType;
}

public FileMetadata(ResourceDescription resource, long contentLength, String contentType) {
super(resource);
public FileMetadata(
ResourceDescription resource, long contentLength, String contentType, Set<ResourceAccessType> permissions) {
super(resource, permissions);
this.contentLength = contentLength;
this.contentType = contentType;
}
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/com/epam/aidial/core/data/Invitation.java
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
@NoArgsConstructor
public class Invitation {
String id;
Set<ResourceLink> resources;
Set<SharedResource> resources;
long createdAt;
long expireAt;
}
5 changes: 5 additions & 0 deletions src/main/java/com/epam/aidial/core/data/MetadataBase.java
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
package com.epam.aidial.core.data;

import com.fasterxml.jackson.annotation.JsonInclude;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

import java.util.Set;

@AllArgsConstructor
@Data
@NoArgsConstructor
Expand All @@ -16,4 +19,6 @@ public abstract class MetadataBase {
private String url;
private NodeType nodeType;
private ResourceType resourceType;
@JsonInclude(JsonInclude.Include.NON_NULL)
private Set<ResourceAccessType> permissions;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
package com.epam.aidial.core.data;

public enum ResourceAccessType {
READ,
WRITE
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import lombok.experimental.Accessors;

import java.util.List;
import java.util.Set;

@Getter
@Setter
Expand All @@ -19,21 +20,45 @@ public class ResourceFolderMetadata extends MetadataBase {
@JsonInclude(JsonInclude.Include.NON_NULL)
private String nextToken;

public ResourceFolderMetadata(ResourceType type, String bucket, String name, String path, String url, List<MetadataBase> items) {
super(name, path, bucket, url, NodeType.FOLDER, type);
public ResourceFolderMetadata(
ResourceType type,
String bucket,
String name,
String path,
String url,
Set<ResourceAccessType> permissions,
List<MetadataBase> items) {
super(name, path, bucket, url, NodeType.FOLDER, type, permissions);
this.items = items;
}

public ResourceFolderMetadata(ResourceType type, String bucket, String name, String path, String url) {
this(type, bucket, name, path, url, null);
public ResourceFolderMetadata(
ResourceType type,
String bucket,
String name,
String path,
String url,
Set<ResourceAccessType> permissions) {
this(type, bucket, name, path, url, permissions, null);
}

public ResourceFolderMetadata(ResourceDescription resource) {
this(resource, null, null);
public ResourceFolderMetadata(ResourceDescription resource, Set<ResourceAccessType> permissions) {
this(resource, permissions, null, null);
}

public ResourceFolderMetadata(ResourceDescription resource, List<MetadataBase> items, String nextToken) {
this(resource.getType(), resource.getBucketName(), resource.getName(), resource.getParentPath(), resource.getUrl(), items);
public ResourceFolderMetadata(
ResourceDescription resource,
Set<ResourceAccessType> permissions,
List<MetadataBase> items,
String nextToken) {
this(
resource.getType(),
resource.getBucketName(),
resource.getName(),
resource.getParentPath(),
resource.getUrl(),
permissions,
items);
this.nextToken = nextToken;
}
}
Loading
Loading