Skip to content

Commit

Permalink
Only copy large binary files
Browse files Browse the repository at this point in the history
Summary: If an input to `glean merge` is already larger than the specified max merged file size, it will just copy it to the output. If the input is in json format, then this causes merge to output json which is unexpected. Only perform the copy if the input is in binary format.

Reviewed By: malanka

Differential Revision: D60988559

fbshipit-source-id: a8e1ffbe1d152ace95def3884d1d53896d016bef
  • Loading branch information
darichey authored and facebook-github-bot committed Aug 9, 2024
1 parent ccc2fff commit 3a48fc8
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 1 deletion.
1 change: 1 addition & 0 deletions glean/tools/gleancli/GleanCLI/Common.hs
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ handleOpt = textOption
data FileFormat
= JsonFormat
| BinaryFormat
deriving (Eq)

instance Show FileFormat where
show ff = case ff of
Expand Down
3 changes: 2 additions & 1 deletion glean/tools/gleancli/GleanCLI/Merge.hs
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,8 @@ instance Plugin MergeCommand where
loop !n _ (Just set) [] = write (n, Right set)
loop !n currentSize acc (f : files) = do
size <- fromIntegral <$> getFileSize f
if size > mergeFileSize -- just copy huge files
if size > mergeFileSize && fileFormat == BinaryFormat
-- just copy huge binary files
then do
write (n, Left f)
loop (n+1) currentSize acc files
Expand Down

0 comments on commit 3a48fc8

Please sign in to comment.