File System

This wraps the fs2 Files API to create a BinaryStore.

Usage

You need a FsStoreConfig object to create an instance of the FsBinaryStore. The companion object has some convenience constructors.

The FsBinaryStore.default creates a store that saves files in a subdirectory hierarchy.

import binny._
import binny.fs._
import fs2.io.file.{Files, Path}
import cats.effect.IO
import cats.effect.unsafe.implicits._
import fs2.Stream

val logger = binny.util.Logger.silent[IO]
// logger: util.Logger[IO] = binny.util.Logger$$anon$2@52ec5ba6
val someData = ExampleData.file2M
// someData: Binary[IO] = Stream(..)

// lets store two pieces and look at the outcome
val run =
  for {
    baseDir <- Stream.resource(DocUtil.tempDir)
    store = FsBinaryStore.default(logger, baseDir)
    id1 <- someData.through(store.insert)
    id2 <- someData.through(store.insert)
    layout <- Stream.eval(DocUtil.directoryContentAsString(baseDir))
  } yield (id1, id2, layout)
// run: Stream[[x]IO[x], (BinaryId, BinaryId, String)] = Stream(..)

run.compile.lastOrError.unsafeRunSync()
// res0: (BinaryId, BinaryId, String) = (
//   BinaryId(H8QZY941vLgEaeCpMBEf9LxiiexU7FF8YDZRTyLgJSo3),
//   BinaryId(3HNfwAqd6RKoHzH9524fP4tsJh9uYVy3tteUfreA6qqN),
//   """
// 📁 binny-docs-3363400676315580922
//   📁 3H
//     📁 3HNfwAqd6RKoHzH9524fP4tsJh9uYVy3tteUfreA6qqN
//       · file
//   📁 H8
//     📁 H8QZY941vLgEaeCpMBEf9LxiiexU7FF8YDZRTyLgJSo3
//       · file"""
// )

This store uses the id to create a directory using the first two characters and another below using the complete id. Then the data is stored in file and its attributes in attr.

This can be changed by providing a different FsStoreConfig. The mapping of an id to a file in the filesystem is given by a PathMapping. There are some provided, the above results are from PathMapping.subdir2.

As another example, the next FsBinaryStore puts the files directly into the baseDir - using the id as its name.

val run2 =
  Stream.resource(DocUtil.tempDir).flatMap { baseDir =>
    val store = FsBinaryStore[IO](
      FsStoreConfig.default(baseDir).withMapping(PathMapping.simple),
      logger
    )
    someData.through(store.insertWith(BinaryId("hello-world.txt"))) ++
      someData.through(store.insertWith(BinaryId("hello_world.txt"))) ++
      Stream.eval(DocUtil.directoryContentAsString(baseDir))
  }
// run2: Stream[[x]IO[x], String] = Stream(..)

run2.compile.lastOrError.unsafeRunSync()
// res1: String = """
// 📁 binny-docs-10763800237879449321
//   · hello-world.txt
//   · hello_world.txt"""

A PathMapping is a function (Path, BinaryId) => Path) where the given path is the base directory. So you can easily create a custom layout.

FsChunkedBinaryStore

The FsChunkedBinaryStore implements ChunkedBinaryStore to allow storing chunks independently. This is useful if chunks are received in random order and the whole file is not available as complete stream.

This is implemented by storing each chunk as a file and concatenating these when loading. Therefore, a DirectoryMapping is required that maps a BinaryId to a directory (and not a file as PathMapping does). For binaries that are provided as a complete stream, it stores just one chunk file - same as FsBinaryStore does.

However, in order to use this the complete size of the file must be known up front. This is needed to know when the last chunk is received.

FsBinaryStoreWithCleanup

The FsBinaryStoreWithCleanup is a wrapper around a FsBinaryStore that upon deletion of a file also removes all empty directories until its base path. Depending on which PathMapping is used, deleting a file could leave empty directories behind. The reason for this default behavior is so inserting and deleting are independent as they won’t write into same subdirectories. The FsBinaryStoreWithCleanup makes sure that inserting and deleting happen interleaved.