MinIO

MiniO is a S3 compatible object storage which can be used to back a BinaryStore. It is available under a free license and can be self hosted. This module provides an implementation that is based on MinIOs Java SDK.

The SDK provides an easy to use asynchronous http client. It is wrapped in an Async.

Usage

To create such a store, you only need the endpoint url, the credentials (an access- and secret key) and a mapping from a BinaryId to a pair of values: S3Key. The MinIO store expects its key to have a bucket and a filekey (or object id). Since a BinaryId is just one value, the store must know how to create the two values, bucket name and object id - which are combined in a S3Key.

The example below starts a container running MinIO in order to work. The container knows the endpoint and keys and can create a config for creating the store. It’s only necessary to specify the mentioned mapping. Here we use a constant bucket name docs-bucket.

import binny._
import binny.minio._
import binny.util.Logger
import ExampleData._
import fs2.Stream
import cats.effect.IO
import cats.effect.unsafe.implicits._

val logger = Logger.silent[IO]
// logger: Logger[IO] = binny.util.Logger$$anon$2@720f29f0
val someData = ExampleData.file2M
// someData: Binary[IO] = Stream(..)

// Create the `BinaryStore`
val store = MinioBinaryStore[IO](DocUtil.minioConfig, logger)
// store: MinioBinaryStore[IO] = binny.minio.MinioBinaryStore@7b3fd68e

val run =
  for {
    // insert some data
    id <- someData.through(store.insert)

    // get the file out
    bin <- Stream.eval(
      store.findBinary(id, ByteRange(0, 100)).getOrElse(sys.error("not found"))
    )
    str <- Stream.eval(bin.readUtf8String)
  } yield str + "..."
// run: Stream[[x]IO[x], String] = Stream(..)

run.compile.lastOrError.unsafeRunSync()
// res0: String = """hello world 1
// hello world 2
// hello world 3
// hello world 4
// hello world 5
// hello world 6
// hello world 7
// he..."""

MinioChunkedBinaryStore

The MinioChunkedBinaryStore implements ChunkedBinaryStore to allow uploading files in independent chunks. This is useful if chunks are received in random order and the whole file is not available as complete stream.

For this the given S3KeyMapping is amended: the bucket is reused as is, but the object-name is appended with /chunk_00001 so the user given objectname is turned into a folder and each chunk is stored beneath. For binaries that are provided as a complete stream, it stores just one chunk file - similar to what MinioBinaryStore does.

When a file is requested, all required chunks are loaded sequentially and concatenated.