Skip to content

Commit

Permalink
Merge pull request #1790 from alcaeus/gridfs-2.0
Browse files Browse the repository at this point in the history 
[2.0] Add GridFS implementation on top of mongodb/mongodb
  • Loading branch information
@alcaeus
alcaeus authored Aug 3, 2018
2 parents f0fe3a6 + f00a3c9 commit 13ea997
Show file tree
Hide file tree
Showing 44 changed files with 1,773 additions and 280 deletions.
2 changes: 1 addition & 1 deletion composer.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
"symfony/console": "~2.3|~3.0|^4.0",
"doctrine/annotations": "~1.2",
"doctrine/collections": "~1.1",
"doctrine/common": "~2.4",
"doctrine/common": "~2.6",
"doctrine/cache": "~1.0",
"doctrine/instantiator": "^1.0.1",
"mongodb/mongodb": "^1.2.0"
Expand Down
268 changes: 181 additions & 87 deletions docs/en/reference/annotations-reference.rst
View file

Large diffs are not rendered by default.

275 changes: 166 additions & 109 deletions docs/en/reference/storing-files-with-mongogridfs.rst
View file
Original file line number Diff line number Diff line change
@@ -1,190 +1,247 @@
Storing Files with MongoGridFS
==============================

The PHP Mongo extension provides a nice and convenient way to store
files in chunks of data with the
`MongoGridFS <http://us.php.net/manual/en/class.mongogridfs.php>`_.

It uses two database collections, one to store the metadata for the
file, and another to store the contents of the file. The contents
are stored in chunks to avoid going over the maximum allowed size
of a MongoDB document.

You can easily setup a Document that is stored using the
MongoGridFS:
Storing Files with GridFS
=========================

About GridFS
------------

With GridFS, MongoDB provides a specification for storing and retrieving files
that exceed the document size limit of 16 MB. GridFS uses two collections to
store files. One collection stores the file chunks, and the other stores file
metadata. More information on GridFS can be found in the
`MongoDB GridFS documentation <https://docs.mongodb.com/manual/core/gridfs/>`_.

GridFS files provide the following properties
-
``_id`` stores the identifier of the file. By default, it uses a BSON
ObjectId, although you can override this in the mapping.
-
``chunkSize`` stores the size of a single chunk in bytes. By default, chunks
are 261120 bytes (i.e. 255 KiB) in size.
-
``filename`` is the name of the file as assigned. Note that filenames don't
need to be unique: instead, multiple files with the same name are treated
as revisions of that same file, with the last file uploaded being the latest
revision.
-
``length`` stores the size of the file in bytes.
-
``metadata`` is an optional embedded document that can be used to store
additional data along with the file.
-
``uploadDate`` stores the date when the file was originally persisted to the
GridFS bucket. It is also used to track revisions of multiple files with the
same filename.

Mapping documents as GridFS files
---------------------------------

.. code-block:: php
<?php
namespace Documents;
/** @Document */
use Doctrine\ODM\MongoDB\Mapping\Annotations\File;
use Doctrine\ODM\MongoDB\Mapping\Annotations\Id;
/** @File(bucketName='image') */
class Image
{
/** @Id */
private $id;
/** @Field */
/** @File\Filename */
private $name;
/** @File */
private $file;
/** @Field */
/** @File\UploadDate */
private $uploadDate;
/** @Field */
/** @File\Length */
private $length;
/** @Field */
/** @File\ChunkSize */
private $chunkSize;
/** @Field */
private $md5;
/** @File\Metadata(targetDocument=ImageMetadata::class) */
private $metadata;
public function getId(): ?string
{
return $this->id;
}
public function setName(string $name): void
public function getName(): ?string
{
return $this->name;
}
public function getChunkSize(): ?int
{
$this->name = $name;
return $this->chunkSize;
}
public function getName(): ?string
public function getLength(): ?int
{
return $this->name;
return $this->length;
}
public function getFile(): ?string
public function getUploadDate(): \DateTimeInterface
{
return $this->file;
return $this->uploadDate;
}
public function setFile(string $file): void
public function getMetadata(): ?ImageMetadata
{
$this->file = $file;
return $this->metadata;
}
}
Notice how we annotated the $file property with @File. This is what
tells the Document that it is to be stored using the MongoGridFS
and the MongoGridFSFile instance is placed in the $file property
for you to access the actual file itself.
If you would rather use XML to map metadata, the corresponding mapping would
look like this:

The $uploadDate, $chunkSize and $md5 properties are automatically filled in
for each file stored in GridFS (whether you like that or not).
Feel free to create getters in your document to actually make use of them,
but keep in mind that their values will be initially unset for new objects
until the next time the document is hydrated (fetched from the database).
..code-block:: xml

First you need to create a new Image:
<?xml version="1.0" encoding="UTF-8"?>

.. code-block:: php
<doctrine-mongo-mapping xmlns="http://doctrine-project.org/schemas/odm/doctrine-mongo-mapping"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://doctrine-project.org/schemas/odm/doctrine-mongo-mapping
http://doctrine-project.org/schemas/odm/doctrine-mongo-mapping.xsd">

<?php
<gridfs-file name="Documents\Image">
<id />
<length />
<chunk-size />
<upload-date />
<filename fieldName="name" />

$image = new Image();
$image->setName('Test image');
$image->setFile('/path/to/image.png');
<metadata target-document="Documents\ImageMetadata" />
</gridfs-file>
</doctrine-mongo-mapping>

$dm->persist($image);
$dm->flush();
With XML mappings, the fields are automatically mapped to camel-cased properties.
To change property names, simply override the ``fieldName`` attribute for each
field. You cannot override any other options for GridFS fields.

Now you can later query for the Image and render it:
The ``ImageMetadata`` class must be an embedded document:

.. code-block:: php
..code-block:: php

<?php

$image = $dm->createQueryBuilder('Documents\Image')
->field('name')->equals('Test image')
->getQuery()
->getSingleResult();
namespace Documents;

header('Content-type: image/png;');
echo $image->getFile()->getBytes();
use Doctrine\ODM\MongoDB\Mapping\Annotations\EmbeddedDocument;
use Doctrine\ODM\MongoDB\Mapping\Annotations\Field;

You can of course make references to this Image document from
another document. Imagine you had a Profile document and you wanted
every Profile to have a profile image:
/** @EmbeddedDocument */
class ImageMetadata
{
/** @Field(type="string") */
private $contentType;
public function __construct(string $contentType)
{
$this->contentType = $contentType;
}

public function getContentType(): ?string
{
return $this->contentType;
}
}

Inserting files into GridFS buckets
-----------------------------------

To insert a new file, you have to upload its contents using the repository. You
have the option to upload contents from a file or a stream. Alternatively, you
can also open an upload stream and write contents yourself.

.. code-block:: php
<?php
namespace Documents;
$repository = $documentManager->getRepository(Documents\Image::class);
$file = $repository->uploadFromFile('image.jpg', '/tmp/path/to/image');
/** @Document */
class Profile
{
/** @Id */
private $id;
When using the default GridFS repository implementation, the ``uploadFromFile``
and ``uploadFromStream`` methods return a proxy object of the file you just
uploaded.

/** @Field */
private $name;
If you want to pass options, such as a metadata object to the uploaded file, you
can pass an ``UploadOptions`` object as the last argument to the
``uploadFromFile``, ``uploadFromStream``, or ``openUploadStream`` method call:

/** @ReferenceOne(targetDocument="Documents\Image") */
private $image;
.. code-block:: php
public function getId(): ?string
{
return $this->id;
}
<?php
public function getName(): ?string
{
return $this->name;
}
use Doctrine\ODM\MongoDB\Repository\UploadOptions;
public function setName(string $name): void
{
$this->name = $name;
}
$uploadOptions = new UploadOptions();
$uploadOptions->metadata = new Documents\ImageMetadata('image/jpeg');
$uploadOptions->chunkSizeBytes = 1024 * 1024;
public function getImage(): ?Image
{
return $this->image;
}
$repository = $documentManager->getRepository(Documents\Image::class);
$file = $repository->uploadFromFile('image.jpg', '/tmp/path/to/image', $uploadOptions);
public function setImage(Image $image): void
{
$this->image = $image;
}
}
Reading files from GridFS buckets
---------------------------------

Now you can create a new Profile and give it an Image:
When reading GridFS files, they behave like all other documents. You can query
for them using the ``find*`` methods in the repository, create query or
aggregation pipeline builders, and also use them as ``targetDocument`` in
references. You can access all properties of the file including metadata, but
not file content.

The GridFS specification uses streams to deal with file contents. To avoid
having this resource overhead every time you fetch a file from the database,
file contents are only provided through the ``downloadToStream`` repository
method. Accessors to provide a stream in the document may be implemented in
future versions.

The following code sample puts the file contents into a different file after
uploading:

.. code-block:: php
<?php
$image = new Image();
$image->setName('Test image');
$image->setFile('/path/to/image.png');
$repository = $documentManager->getRepository(Documents\Image::class);
$file = $repository->uploadFromFile('image.jpg', '/tmp/path/to/image', new Documents\ImageMetadata('image/jpeg'));
$profile = new Profile();
$profile->setName('Jonathan H. Wage');
$profile->setImage($image);
$stream = fopen('tmp/path/to/copy', 'w+');
try {
$repository->downloadToStream($file->getId(), $stream);
finally {
fclose($stream);
}
$dm->persist($profile);
$dm->flush();
The ``downloadToStream`` method takes the identifier of a file as first argument
and a writable stream as the second arguments. If you need to manipulate the
file contents before writing it to disk or sending it to the client, consider
using a memory stream using the ``php://memory`` stream wrapper.

If you want to query for the Profile and load the Image reference
in a query you can use:
Alternatively, you can also use the ``openDownloadStream`` method which returns
a stream from where you can read file contents:

.. code-block:: php
<?php
$profile = $dm->createQueryBuilder('Profile')
->field('name')->equals('Jonathan H. Wage')
->getQuery()
->getSingleResult();
use Doctrine\ODM\MongoDB\Repository\UploadOptions;
$image = $profile->getImage();
$uploadOptions = new UploadOptions();
$uploadOptions->metadata = new Documents\ImageMetadata('image/jpeg');
$repository = $documentManager->getRepository(Documents\Image::class);
$file = $repository->uploadFromFile('image.jpg', '/tmp/path/to/image', $uploadOptions);
$stream = $repository->openDownloadStream($file->getId());
try {
$contents = stream_get_contents($stream);
finally {
fclose($stream);
}
header('Content-type: image/png;');
echo $image->getFile()->getBytes();
Loading

0 comments on commit 13ea997

Please sign in to comment.