cayley v0.7.0 Release Notes
Release Date: 2017-12-24 // over 6 years ago-
๐ Release covers nearly 8 months of work, mostly on implementing meta-backends, new query optimizer, performance improvements and bug fixes.
Note: From now on we plan to release point releases on regular basis to push fixes and new features faster.
โฌ๏ธ Storage format for backends received an upgrade bringing backward incompatible changes. Check migration guide for more details. It only affects the database format, HTTP API remains backward-compatible.
๐ This release defines new internal interfaces (meta-backends) for working with various database kinds. Currently three of them are defined: KV, NoSQL, SQL. Each of them significantly simplifies a process of implementing a new backend, as well as allows us to lower maintenance overhead by sharing most optimizations, read and write paths, etc. This also means that optimization work on one backend will be automatically applied to other backends of the same kind in the future.
Part of this effort was an implementation of new query representation that we call "shapes". All optimizations like query rewrites and reordering are now done on a generic level, without any backend-specific code. It will allow us to save query stats, make more complex optimizations and even store query templates in the database, or split it to be executed in distributed manner by multiple backends. Since new representation can be manipulated more easily, most backends will now rewrite complex query types into their native query language (SQL, for example).
Highlights
- ๐ ElasticSearch support
- ๐ New query optimizer (aka query shapes)
- ๐ New command line interface
- KV meta-backend + port of Bolt and LevelDB to it
- NoSQL meta-backend, + port of Mongo and ElasticSearch to it
- ๐ "Raw" value type is deprecated
- ๐ฆ Exposed node delete API for HTTP
- Consistent behavior for all backends on node delete
- ๐ New query types in Gizmo:
Labels
,SaveInPredicates
,SaveOutPredicates
, see docs - ๐
Has
traversal now supports value filters FollowRecursive
can be limited to a maximal depth- ๐ GraphQL now supports quad labels
- โก๏ธ SQL query optimizer improvements
- SQL backends now has their own names:
postgres
,mysql
,cockroach
(no need to specifyflavor
) - ๐ Significant performance improvements
- ๐ณ Smaller Docker image
- ๐ป Minor UI improvements
๐ฅ Breaking changes
- Bolt, LevelDB and Mongo backends were rewritten and database format has changed. Old implementations are still available under
bolt1
,leveldb1
andmongo1
names. Old backends will be removed in next minor release. c266392 - SQL table structure has changed 31b0f48, #617
- 0๏ธโฃ Cayley now ignores duplicated quads by default 9cb0eef
- ๐ Gremlin is deprecated and dropped in favor of Gizmo #519
- Count in gizmo is now a 'final' instead of traversal and returns result #576
๐ฅ Breaking changes (Go)
๐ As always, we expect users to pin a specific Cayley version tag with a dependency manager and sync with following changes in the code:
- Cayley uses Go version 1.9+
- Bolt, LevelDB and Mongo backends were rewritten and database format has changed. Old implementations are still available under old paths, but will be dropped in next minor release. Users are encouraged to update imports to
kv/bolt
,kv/leveldb
andnosql/mongo
. - SQL flavors moved to their own packages and needs to be imported separately (
sql/postgres
for example).sql
backend name will be deprecated in next major release. 48e7946 - ๐ "Raw" value type (
quad.Raw
andquad.MakeRaw
) is deprecated and replaced with a helper to preserve at least some compatibility. It's strongly recommended to switch toquad.String
orIRI
andquad.Make
. 41bf496 - ๐ Iterator optimizations for backends are mostly disabled. It is recommended to build queries with new
graph/shape
package and run optimizations on it. - โ Removed unused methods from
QuadStore
:Horizon
,FixedIterator
,Type
d994e2a, b63faea, #631 - โ Removed
graph.PrimaryKey
type d994e2a graph.Value
is now type-safe e4ab526- โ Removed
ID
andTimestamp
fields fromgraph.Delta
- they will be assigned by server 524c8e2 - ๐
RemoveNode
type changed toquad.Value
to match other methods #607 - ๐ Quad writers produced by
graph.NewWriter
andNewRemover
now requiresFlush
to be called a420e62 Iterator
now accepts context forNext
,NextPath
andContains
f3e814f- ๐
Describe
method was removed fromIterator
and replaced byString
method on iterator andgraph.DescribeIterator
b491f41 - Iterator type is now a string instead of enum and registration is not necessary anymore
๐ Changelog
dennwc: http: expose node delete api; resolves #622 dennwc: deprecate quad.Raw type dennwc: code-wide cleanup dennwc: iterator: remove unused equality function for fixed dennwc: graph: remove unused horizon function dennwc: memstore: use correct size for all iterator dennwc: add context argument to iterators and kv dennwc: kv: track nodes reference count; fixes #617 dennwc: sql: track nodes reference count and simplify dialects; fixes #617 dennwc: memstore: remove nodes if all their references are gone; addresses #617 dennwc: iterator: returns correct subiterator slice for empty And; fixes #659 dennwc: nosql: use base64 instead of hex encoding for hashes dennwc: update Go version to 1.9 dennwc: update integration test, make it a part of suite; move paths as well dennwc: fix tests for predicate save query; fix node tokens for bolt dennwc: nosql: use shapes optimizer for queries dennwc: shape: allow to simplify shapes; add helpers for building node filters dennwc: elastic: port to nosql layer; resolves #653 dennwc: nosql: store each typed value in separate field; support batch inserts dennwc: separate mongo logic and define a generic nosql layer; resolves #652 dennwc: iterator: replace Describe method with a helper dennwc: graphql: support @label directive; resolves #614 dennwc: path/gizmo: support Has with filters; resolves #650 dennwc: bolt,leveldb: swap names to a new kv backend implementations michaelqiu94: ElasticSearch as Supported Backend (#634) dennwc: path,gizmo: allow to save predicates without moving from nodes; fix #591 dennwc: http: allow to serve API without web files; fix #503 dennwc: ui: format json results; resolve #454 dennwc: sql: optimize intersection, value lookup and comparison dennwc: sql: fix params propagation from sub-queries dennwc: sql: rewrite query builder to shapes; fix #563 dennwc: shape: optimize multiple Pages; push Fixed values from Intersect into NodesFrom dennwc: propagate sub-iterator errors from And properly dennwc: stream ops in node delete; return error if node is missing; fix #608 dennwc: add a single package that imports all supported backends derekrliang: Add optional max recursion depth to FollowRecursive (#639) derekrliang: Fix query shape error when query has no result data links (#641) derekrliang: Add running local unit test documentation (#640) dennwc: bolt,leveldb: port to kv layer dennwc: kv: add in-memory implementation dennwc: integration: open and close db for each test set; use table benchmarks dennwc: kv: support for index scans and indexes on multiple directions dennwc: kv: allow to customize indexes dennwc: kv: batch get requests and simplify bucket interface Yannic: Reduce Docker image size (#637) dennwc: sql: separate packages for flavors (#633) Yannic: Remove unused Type function (#631) dennwc: docs: add http api definitions (swagger v3) dennwc: iterator: remove duplicated TagResults code and type registration dennwc: github: simplify issue template header, add slack link dennwc: tag values in recursive iterator correctly dennwc: shape: new query optimizer dennwc: memstore: iterator optimizations dennwc: graph: rewrite pk; remove horizon and ts from delta; rewrite memstore dennwc: kv: registration, scan by prefix, support for flat kv dennwc: integration: allow to load data to remote backends dennwc: gizmo: fix #620 and bump goja version dennwc: graph: helper for writing quads to tx (useful for schema) dennwc: writer: add a direct constructor for single replication Jason Kingsbury: make bloom filter buffer before initialisation dennwc: cli: read query timeout from config file; fix #613 dennwc: graphql: allow full IRI charset in names; resolve #616 dennwc: schema: allow to limit depth; fix #609 joostverdoorn: Set CORS headers on gephi stream dennwc: http: allow to specify query limit; fix #612 joostverdoorn: Add labels morphism and Gizmo API (#610) dennwc: http: do not add .md extension if it's already present; fix #597 dennwc: schema: return an error on write if required field is not set; fix #602 dennwc: accept quad.Value instead of graph.Value for RemoveNode; fix #607 dennwc: print warning on set verbosity for glog; fix #599 dennwc: flush writer in schema example; fix #606 harlantwood: Remove "angry highlighting" from JSON examples dennwc: generic kv backend based on bolt h4ck3rm1k3: query from stdin (#601) chrispassas: Fix example startup command dennwc: make graph.Value safe again h4ck3rm1k3: adding help to repl dennwc: schema: handle id in anonymous struct fields h4ck3rm1k3: Update sidebar.tmpl (#595) dennwc: schema: allow to load objects from path dennwc: cli: implement query command barakmich: conversion check h4ck3rm1k3: Update transaction.go dennwc: graph: batch quads automatically if caller uses single AddQuad on writer gkontos: gae: fix add of previously deleted quads (#580) MarioAriasGa: Glog issues (#579) barakmich: bolt2: use bloom filter to speed insertion barakmich: bolt2: working queries barakmich: graph: Add primitive proto barakmich: Fix glide story; ignore docker deps, `glide.lock` is now sane dennwc: gizmo: make Count a final instead of traversal dennwc: fix contains on recursive iterator dennwc: check for nil result from NameOf; fix #574 dennwc: ignore duplicate quads by default dennwc: docs: add example configs for kubernetes; resolves #327 dennwc: do not load test dataset in container dennwc: golang client for data import/export; add tests for v2 api dennwc: http: support queries in api v2; expose api v2 as a library dennwc: http: get rid of custom handler type dennwc: http: use optional interface to create qs for request dennwc: bind to localhost by default owulveryck: The -host option was missing the port argument mikaelcabot: cockroach: schema changes within transactions are restricted (#567) dennwc: gizmo: generate docs from Go function comments; resolve #520 (#560) dennwc: cli: register glog flags; make conv an alias for convert; resolve #561 oren: Update README.md - add MySQL and CockroachDB dennwc: deprecate gremlin in favor of gizmo; resolve #519 dennwc: cli: deduplication tool for nodes with same properties dennwc: cayley: switch to cobra/viper cli