pile/README.md

TODO

• redo sidecars

  • sidecar manipulation commands:
    • fill (defaults, defined in toml)
    • delete
  • writing to files would be nice, but we want to be able to revert changes
  • make sure to upload sidecars
  • json extractor

• incremental index

• better errors (s3 not found)

later

• search ui (betalupi books + handouts)

• lists

• export

• libgen search?

• remote encryption

• publish (arch, nix, crates)

• extractors

  • ogg
  • regex (from any string, filename, etc)
  • whisper (speech-to-text)
  • ocr (pdf pages, pass through fn!)
  • list, fields on schemas instead of fields
  • blobs as items or items as blobs?
  • default args for each label (to_json, count)?
  • which fields to include in json?
  • nonempty (array)

• redo docs

  • source types
  • sidecars (rename?)
  • doc all keys