spaGO v0.5.0 Release Notes

Release Date: 2021-02-15 // about 3 years ago
  • ➕ Added

    • Implement a beam-search algorithm for conditional generation:
      • nlp.transformer.generation package.
    • ➕ Add implementation of the Sentence-Piece tokenizer:
      • nlp.tokenizers.sentencepiece package.
    • BART improvements:
      • gRPC and HTTP API to perform Text Generation.
      • Add support for "Marian" architecture (used for translation tasks).
      • Add sinusoidal positional encoder (used by Marian).
      • Add "head" for conditional generation:
      • nlp.transformers.bart.head.conditionalgeneration package.
    • ➕ Add nn.Closer interface (e.g. embeddings.Model needs to close the underlying key-value store).
    • ➕ Add Swish act. function without trainable parameters.
    • ➕ Add SiLU act. function (it is just an alias for Swish).
    • 🆕 New pe.SinusoidalPositionalEncoder (this implementation replaces unused pe.PositionalEncoder and pe.AxialPositionalEncoder)

    🔄 Changed

    • ⚡️ Update urfave/cli to v2
    • ⚡️ Update dgraph-io/badger to v3.
    • 👉 Make the BART positional encoder an interface to support various encoding (i.e. trainable vs static).
    • Rename to fn.NewSwish into fn.NewSwishB as this was the Swish variant with trainable parameters (B).
    • 😌 Relax ag.GetOpName to match operator names in lower-case.
    • 👍 Allow arbitrary activation function on BART encoder/decoder layers.
    • 👉 Use precomputed "keys" and "values" in self-attention, multi-head attention and BART decoder.

    ✂ Removed

    • In relation to the aforementioned positional encoding changes:
      • pe.PositionalEncoder and related functions
      • pe.AxialPositionalEncoder and related functions

    🛠 Fixed

    • 🛠 Fix causal-mask used by nn.ScaledDotProductAttention