cayley v0.7.0 Release Notes

Release Date: 2017-12-24 // over 2 years ago
  • 🚀 Release covers nearly 8 months of work, mostly on implementing meta-backends, new query optimizer, performance improvements and bug fixes.

    Note: From now on we plan to release point releases on regular basis to push fixes and new features faster.

    ⬆️ Storage format for backends received an upgrade bringing backward incompatible changes. Check migration guide for more details. It only affects the database format, HTTP API remains backward-compatible.

    🚀 This release defines new internal interfaces (meta-backends) for working with various database kinds. Currently three of them are defined: KV, NoSQL, SQL. Each of them significantly simplifies a process of implementing a new backend, as well as allows us to lower maintenance overhead by sharing most optimizations, read and write paths, etc. This also means that optimization work on one backend will be automatically applied to other backends of the same kind in the future.

    Part of this effort was an implementation of new query representation that we call "shapes". All optimizations like query rewrites and reordering are now done on a generic level, without any backend-specific code. It will allow us to save query stats, make more complex optimizations and even store query templates in the database, or split it to be executed in distributed manner by multiple backends. Since new representation can be manipulated more easily, most backends will now rewrite complex query types into their native query language (SQL, for example).

    Highlights

    • 👍 ElasticSearch support
    • 🆕 New query optimizer (aka query shapes)
    • 🆕 New command line interface
    • KV meta-backend + port of Bolt and LevelDB to it
    • NoSQL meta-backend, + port of Mongo and ElasticSearch to it
    • 🗄 "Raw" value type is deprecated
    • 🔦 Exposed node delete API for HTTP
    • Consistent behavior for all backends on node delete
    • 🆕 New query types in Gizmo: Labels, SaveInPredicates, SaveOutPredicates, see docs
    • 👍 Has traversal now supports value filters
    • FollowRecursive can be limited to a maximal depth
    • 👍 GraphQL now supports quad labels
    • ⚡️ SQL query optimizer improvements
    • SQL backends now has their own names: postgres, mysql, cockroach (no need to specify flavor)
    • 🐎 Significant performance improvements
    • 🐳 Smaller Docker image
    • 💻 Minor UI improvements

    💥 Breaking changes

    • Bolt, LevelDB and Mongo backends were rewritten and database format has changed. Old implementations are still available under bolt1, leveldb1 and mongo1 names. Old backends will be removed in next minor release. c266392
    • SQL table structure has changed 31b0f48, #617
    • 0️⃣ Cayley now ignores duplicated quads by default 9cb0eef
    • 🗄 Gremlin is deprecated and dropped in favor of Gizmo #519
    • Count in gizmo is now a 'final' instead of traversal and returns result #576

    💥 Breaking changes (Go)

    🔀 As always, we expect users to pin a specific Cayley version tag with a dependency manager and sync with following changes in the code:

    • Cayley uses Go version 1.9+
    • Bolt, LevelDB and Mongo backends were rewritten and database format has changed. Old implementations are still available under old paths, but will be dropped in next minor release. Users are encouraged to update imports to kv/bolt, kv/leveldb and nosql/mongo.
    • SQL flavors moved to their own packages and needs to be imported separately (sql/postgres for example). sql backend name will be deprecated in next major release. 48e7946
    • 🗄 "Raw" value type (quad.Raw and quad.MakeRaw) is deprecated and replaced with a helper to preserve at least some compatibility. It's strongly recommended to switch to quad.String or IRI and quad.Make. 41bf496
    • 🏗 Iterator optimizations for backends are mostly disabled. It is recommended to build queries with new graph/shape package and run optimizations on it.
    • ✂ Removed unused methods from QuadStore: Horizon, FixedIterator, Type d994e2a, b63faea, #631
    • ✂ Removed graph.PrimaryKey type d994e2a
    • graph.Value is now type-safe e4ab526
    • ✂ Removed ID and Timestamp fields from graph.Delta - they will be assigned by server 524c8e2
    • 🚚 RemoveNode type changed to quad.Value to match other methods #607
    • 🚚 Quad writers produced by graph.NewWriter and NewRemover now requires Flush to be called a420e62
    • Iterator now accepts context for Next, NextPath and Contains f3e814f
    • 🚚 Describe method was removed from Iterator and replaced by String method on iterator and graph.DescribeIterator b491f41
    • Iterator type is now a string instead of enum and registration is not necessary anymore

    🔄 Changelog

    dennwc: http: expose node delete api; resolves #622
    dennwc: deprecate quad.Raw type
    dennwc: code-wide cleanup
    dennwc: iterator: remove unused equality function for fixed
    dennwc: graph: remove unused horizon function
    dennwc: memstore: use correct size for all iterator
    dennwc: add context argument to iterators and kv
    dennwc: kv: track nodes reference count; fixes #617
    dennwc: sql: track nodes reference count and simplify dialects; fixes #617
    dennwc: memstore: remove nodes if all their references are gone; addresses #617
    dennwc: iterator: returns correct subiterator slice for empty And; fixes #659
    dennwc: nosql: use base64 instead of hex encoding for hashes
    dennwc: update Go version to 1.9
    dennwc: update integration test, make it a part of suite; move paths as well
    dennwc: fix tests for predicate save query; fix node tokens for bolt
    dennwc: nosql: use shapes optimizer for queries
    dennwc: shape: allow to simplify shapes; add helpers for building node filters
    dennwc: elastic: port to nosql layer; resolves #653
    dennwc: nosql: store each typed value in separate field; support batch inserts
    dennwc: separate mongo logic and define a generic nosql layer; resolves #652
    dennwc: iterator: replace Describe method with a helper
    dennwc: graphql: support @label directive; resolves #614
    dennwc: path/gizmo: support Has with filters; resolves #650
    dennwc: bolt,leveldb: swap names to a new kv backend implementations
    michaelqiu94: ElasticSearch as Supported Backend (#634)
    dennwc: path,gizmo: allow to save predicates without moving from nodes; fix #591
    dennwc: http: allow to serve API without web files; fix #503
    dennwc: ui: format json results; resolve #454
    dennwc: sql: optimize intersection, value lookup and comparison
    dennwc: sql: fix params propagation from sub-queries
    dennwc: sql: rewrite query builder to shapes; fix #563
    dennwc: shape: optimize multiple Pages; push Fixed values from Intersect into NodesFrom
    dennwc: propagate sub-iterator errors from And properly
    dennwc: stream ops in node delete; return error if node is missing; fix #608
    dennwc: add a single package that imports all supported backends
    derekrliang: Add optional max recursion depth to FollowRecursive (#639)
    derekrliang: Fix query shape error when query has no result data links (#641)
    derekrliang: Add running local unit test documentation (#640)
    dennwc: bolt,leveldb: port to kv layer
    dennwc: kv: add in-memory implementation
    dennwc: integration: open and close db for each test set; use table benchmarks
    dennwc: kv: support for index scans and indexes on multiple directions
    dennwc: kv: allow to customize indexes
    dennwc: kv: batch get requests and simplify bucket interface
    Yannic: Reduce Docker image size (#637)
    dennwc: sql: separate packages for flavors (#633)
    Yannic: Remove unused Type function (#631)
    dennwc: docs: add http api definitions (swagger v3)
    dennwc: iterator: remove duplicated TagResults code and type registration
    dennwc: github: simplify issue template header, add slack link
    dennwc: tag values in recursive iterator correctly
    dennwc: shape: new query optimizer
    dennwc: memstore: iterator optimizations
    dennwc: graph: rewrite pk; remove horizon and ts from delta; rewrite memstore
    dennwc: kv: registration, scan by prefix, support for flat kv
    dennwc: integration: allow to load data to remote backends
    dennwc: gizmo: fix #620 and bump goja version
    dennwc: graph: helper for writing quads to tx (useful for schema)
    dennwc: writer: add a direct constructor for single replication
    Jason Kingsbury: make bloom filter buffer before initialisation
    dennwc: cli: read query timeout from config file; fix #613
    dennwc: graphql: allow full IRI charset in names; resolve #616
    dennwc: schema: allow to limit depth; fix #609
    joostverdoorn: Set CORS headers on gephi stream
    dennwc: http: allow to specify query limit; fix #612
    joostverdoorn: Add labels morphism and Gizmo API (#610)
    dennwc: http: do not add .md extension if it's already present; fix #597
    dennwc: schema: return an error on write if required field is not set; fix #602
    dennwc: accept quad.Value instead of graph.Value for RemoveNode; fix #607
    dennwc: print warning on set verbosity for glog; fix #599
    dennwc: flush writer in schema example; fix #606
    harlantwood: Remove "angry highlighting" from JSON examples
    dennwc: generic kv backend based on bolt
    h4ck3rm1k3: query from stdin (#601)
    chrispassas: Fix example startup command
    dennwc: make graph.Value safe again
    h4ck3rm1k3: adding help to repl
    dennwc: schema: handle id in anonymous struct fields
    h4ck3rm1k3: Update sidebar.tmpl (#595)
    dennwc: schema: allow to load objects from path
    dennwc: cli: implement query command
    barakmich: conversion check
    h4ck3rm1k3: Update transaction.go
    dennwc: graph: batch quads automatically if caller uses single AddQuad on writer
    gkontos: gae: fix add of previously deleted quads (#580)
    MarioAriasGa: Glog issues (#579)
    barakmich: bolt2: use bloom filter to speed insertion
    barakmich: bolt2: working queries
    barakmich: graph: Add primitive proto
    barakmich: Fix glide story; ignore docker deps, `glide.lock` is now sane
    dennwc: gizmo: make Count a final instead of traversal
    dennwc: fix contains on recursive iterator
    dennwc: check for nil result from NameOf; fix #574
    dennwc: ignore duplicate quads by default
    dennwc: docs: add example configs for kubernetes; resolves #327
    dennwc: do not load test dataset in container
    dennwc: golang client for data import/export; add tests for v2 api
    dennwc: http: support queries in api v2; expose api v2 as a library
    dennwc: http: get rid of custom handler type
    dennwc: http: use optional interface to create qs for request
    dennwc: bind to localhost by default
    owulveryck: The -host option was missing the port argument
    mikaelcabot: cockroach: schema changes within transactions are restricted (#567)
    dennwc: gizmo: generate docs from Go function comments; resolve #520 (#560)
    dennwc: cli: register glog flags; make conv an alias for convert; resolve #561
    oren: Update README.md - add MySQL and CockroachDB
    dennwc: deprecate gremlin in favor of gizmo; resolve #519
    dennwc: cli: deduplication tool for nodes with same properties
    dennwc: cayley: switch to cobra/viper cli