spaGO v0.5.0 Release Notes
Release Date: 2021-02-15 // about 3 years ago-
➕ Added
- Implement a beam-search algorithm for conditional generation:
nlp.transformer.generation
package.
- ➕ Add implementation of the Sentence-Piece tokenizer:
nlp.tokenizers.sentencepiece
package.
- BART improvements:
- gRPC and HTTP API to perform Text Generation.
- Add support for "Marian" architecture (used for translation tasks).
- Add sinusoidal positional encoder (used by Marian).
- Add "head" for conditional generation:
nlp.transformers.bart.head.conditionalgeneration
package.
- ➕ Add
nn.Closer
interface (e.g.embeddings.Model
needs to close the underlying key-value store). - ➕ Add Swish act. function without trainable parameters.
- ➕ Add SiLU act. function (it is just an alias for Swish).
- 🆕 New
pe.SinusoidalPositionalEncoder
(this implementation replaces unusedpe.PositionalEncoder
andpe.AxialPositionalEncoder
)
🔄 Changed
- ⚡️ Update urfave/cli to v2
- ⚡️ Update dgraph-io/badger to v3.
- 👉 Make the BART positional encoder an interface to support various encoding (i.e. trainable vs static).
- Rename to
fn.NewSwish
intofn.NewSwishB
as this was the Swish variant with trainable parameters (B). - 😌 Relax
ag.GetOpName
to match operator names in lower-case. - 👍 Allow arbitrary activation function on BART encoder/decoder layers.
- 👉 Use precomputed "keys" and "values" in self-attention, multi-head attention and BART decoder.
✂ Removed
- In relation to the aforementioned positional encoding changes:
pe.PositionalEncoder
and related functionspe.AxialPositionalEncoder
and related functions
🛠 Fixed
- 🛠 Fix causal-mask used by
nn.ScaledDotProductAttention
- Implement a beam-search algorithm for conditional generation: