vitess v7.0.0-beta Release Notes

Release Date: 2020-07-21 // over 3 years ago
  • Incompatible Changes

    ๐Ÿ‘€ VTGate: The default gateway ("-gateway_implementation" flag value) is now "tabletgateway". The old default was "discoverygatewayโ€. To use the old gateway, specify the flag in the command line. Anyone already using the flag will see no change; to use the new gateway, remove the -gateway_implementation flag. The new gateway is simpler and uses a new healthcheck module with no loss of functionality. #6155

    โš  VTGate: VTGate incorrectly returned Warning Code field name as Type. This has now been fixed. Code or scripts that depend on the incorrect behavior might break. #6437

    ๐Ÿ—„ Deprecations

    ๐Ÿš€ The following vtctl commands have been deprecated. Deprecated commands will continue to work until the next release (8.0). #6428

    StartSlave -> Replaced with StartReplication

    StopSlave -> Replaced with StopReplication

    ๐Ÿ”„ ChangeSlaveType -> Replaced with ChangeTabletType

    Various vtctl commands had a flag called "-wait_slave_timeout". It has been replaced with โ€œ-wait_replicas_timeoutโ€. Scripts using the old flag will continue to work until the next major release (8.0). #6428

    Helm chart (more details to be added)

    Consul (more details to be added)

    Replaced VTTablet ExecuteBatch api call with Execute for autocommit query. #6407 #6431

    ๐Ÿ› Bugs Fixed

    VTGate / MySQL compatibility

    VTGate: When using user-defined variables that have not been initialized first, a non-fatal panic occurs and the connection is dropped. #6294 #6308

    VTGate: Setting system variable panics vtgate. #6162 #6163

    regression: USE does not work from mysql client. #6153 #6157

    ๐Ÿ‘‰ USE does not return an error if a keyspace doesn't exist. #3060 #6157

    Dotted table names are not escaped properly in lookup vindex flow. #6142 #6145

    Error 1105 instead of 1062 when inserting duplicate into consistent_lookup vindex. #6144

    VTGate: Rewriting expressions fail when used as function parameters. #6281

    โšก๏ธ Rollback after error clobbers updates. #6285

    Queries should not be wrapped into transactions if autocommit=1 is set. #6264

    Prepared statements with bind variables in column specification produce โ€˜missing bind varโ€™ error. #6287 #6298

    ๐Ÿ“œ Prepared statement queries returning NULL results are not parsed. #6240 #6249

    0๏ธโƒฃ DEFAULT function not working with status column. #6221 #6224

    VTGate panic kills connection when selecting database() when using OLAP workload. #6200 #6205

    Wrong column value can be inserted due to parameterization variable confusion. #6266 #6270

    ๐Ÿ›  Fix owned table name formatting and duplicate error code in consistent lookup. #6145

    ๐Ÿ‘Œ Support for case-insensitive user defined variable. #6225

    ๐Ÿ›  Fix performance regression on VTGate. #6203

    Return NULL for non-existent user-defined variables. #6308

    ๐Ÿ”’ Lock function support. #6370

    VTGate: MySQL connection closing should not inflate query counts. #6226

    ๐Ÿ›  VTGate: Show vitess_shards intermittently returns empty list. This has been fixed #5038 #5189

    Other

    โฑ VTTablet: Etcd query timeout because auto-committable transactions are not properly rolled back. #6258 #6299

    ๐Ÿ›  VTTablet: Fixed race in schema engine during notification of schema changes. #6268

    ๐Ÿ›  VTTablet: Reloading ACL config was throwing misleading error messages.his has now been fixed and the error messages improved. #6309

    ๐Ÿš€ VTTablet: Panic during backup because of an error in the pgzip library has been fixed by contributing a fix to pgzip and upgrading to a new release that includes the fix. #6189

    VTTablet: Improvement of error handling by retrying while uploading to S3. #6182 #6395 #6405

    ๐Ÿ›  VTTablet: builtinbackupengine was uploading a manifest despite errors from S3. Error collection during backup has been fixed to address this. #6349 #6350

    VTTablet: Fix dba connection leak in LockTables. #6424

    VTTablet: Fix connection pool leak in schema engine. #6426

    VTTablet: Fix panic from binlog_streamer when โ€˜-watch_replication_streamโ€™ is set., #6253

    ๐Ÿ’ป Vtctld UI: Change all external links in vitess to use https #6170 #6169

    Ensure SrvVSchema gets rebuilt for all cells #6276

    ๐Ÿ›  Fix MariadbGTIDSet multi-domain support #6184

    mysqlctl: Fix connection leak during killConnection() #6245

    ๐Ÿ›  Fix release script to install examples in the correct directory #6159

    Functionality Added or Changed

    VTGate/ MySQL compatibility

    ๐Ÿ‘ Transactions on read-only replicas are now supported. Start a transaction by first choosing the desired tablet_type ("use @replica"), then issuing a โ€œBEGINโ€ statement. Internally, this will issue a โ€œSTART TRANSACTION READ ONLYโ€ on the underlying database. This is only available if using tabletgateway. #6166 #6244

    ๐Ÿ‘ Create better plans for NULL comparisons. Instead of using a SelectScatter, now vtgate will produce empty results when the predicate is a comparison against literal NULLs. #6133 #6425

    โž• Add EXPLAIN support for vtgate plans. View the produced vtgate query plan by using EXPLAIN FORMAT=VITESS.

    โž• Added savepoint support. #6355 #6404 #6412 #6414

    "ignore_nulls" option to lookup vindexes #6147 #6222

    โž• Add support for row_count() function. #6174 #6179

    ๐Ÿ‘ XOR Operator is now supported in sql WHERE clause. #6369 #6371

    Allow 'binary' for character_set_results. It is equivalent to null. #6237

    ๐Ÿ‘Œ Improve logging by adding keyspace, table, and tablet_type. #6282

    ๐Ÿ›ฐ VTGate now takes in optional flags for โ€˜max_payload_sizeโ€™ and โ€˜warn_payload_sizeโ€™. A payload size above the warning threshold will only produce a warning. A payload size that exceeds โ€˜max_payload_sizeโ€™ will result in an error. The default behavior is not to check payload size, which is backwards-compatible. #6143 #6375

    A new comment directive (โ€˜/*vt+ IGNORE_MAX_PAYLOAD_SIZE=1 */) has been added to provide the ability to override the payload size checking per query. #6143

    Support โ€˜select * from information_schema.processlistโ€™. #6310

    ๐Ÿ‘ VTGate: Support batch lookup queries for integer and binary types. Other types will continue to require one query per row. #6420

    ๐Ÿ‘Œ Support system settings at session level. #6149

    Prepare for future features by adding reserved connections. This is the ability for a connected client to have a dedicated connection instead of using the connection pool. #6303 #6313

    ๐Ÿ‘ Allow charset introducers with vindex column values. #6236 #6314

    ๐Ÿ‘Œ Support Union All between different shards. #6351

    VTGate: Exclude KeyspaceId(s) from plan cache key.This improves the cache hit rate. #6233

    VTGate: Reload TLS config on SIGHUP. #6215

    VTGate: Drop vindex DDL. #6151

    VTGate: Show vitess_tablets now reports master_term_start_time. We ensure that this is reported correctly after reparents. #6135 #6292 #6293

    VReplication

    VStream Copy extends the VStream API to stream entire databases or tables. Earlier you specified a position to start streaming events from. #6277

    0๏ธโƒฃ VReplication workflows now accept "-cells" as an option. This can be a comma-separated list of cells or cellsAliases. The source tablet for vreplication will be chosen randomly among the tablets that belong to the specified cells/cellsAliases. If the option is not provided, we default to the cell that is local to the vreplication target. #6442, #6456

    Schema Versioning. Tablets now have a schema tracker that can be enabled to store schema versions as DDLs are applied. The _vt.schema_version table contains the schema with the corresponding ddl and gtid position when the ddl was applied. The tracked schemas are used by the new Historian service to provide vstreams with the correct schema while streaming old events. #6300

    ๐Ÿ‘ Foreign Keys are now supported on the target database during vreplication. In the copy phase we disable FK checks and re-enable them once the replication phase starts. #6284

    Materialize: added a drop_constraint option during table copy to strip constraints while creating target tables #6422

    ๐ŸŽ Materialize: optimizations while creating target tables resulting in a significant improvement in performance when moving a large number of tables from an external MySQL instance #6207, #6251

    โฌ‡๏ธ DropSources: Added โ€˜-rename_tablesโ€™ option to rename the source tables rather than dropping them entirely. #6383

    โž• Add keyspace, shard and workflow labels to VReplicationSecondsBehindMaster metrics #6279

    ๐Ÿ’ป Correct command line help #6187

    Point-in-time Recovery

    ๐Ÿš€ In the 4.0 release we introduced functionality to recover a keyspace into a snapshot based on a specified time. That feature only allowed you to restore a backup, but did not have the ability to replay binlogs up to the specified time. The feature has now been completed so that true point-in-time recovery is possible. This is currently only supported for MySQL compatible installations, not MariaDB. #4857 #6267 #6408

    Healthcheck

    ๐Ÿ—„ VTGate can now use a new healthcheck that is logically simpler while providing the same functionality. This is the default. The older healthcheck has been deprecated but is still available to use if desired by setting "-gateway_implementation=discoverygateway" #5750 #6155

    VTCombo has been migrated to use the new healthcheck #6302

    MiniVitess

    ๐Ÿณ A docker container that can be used to bring up a self-contained vitess cluster, which can automatically resolve and self-configure to meet an external MySQL cluster. While this might sound very similar to VTCombo and vttestserver, it is not quite the same. MiniVitess runs in docker, brings up distinct processes for each vitess component and works with an external MySQL cluster. #6377

    Examples / Tutorials

    Vitess operator example #6154

    Region sharding example #6209 #6275

    ๐Ÿ”„ Change local example to use new command DropSources #6178

    โœ… Are-you-alive example now makes it easier to test multiple endpoints #6202

    ๐Ÿณ Local docker tutorial added to website

    Other

    โšก๏ธ Vtctl ListAllTablets now shows the MasterTermStartTime. This is shown as "" for read-only tablet types. In case of an old master that has not yet updated the topo record, TabletType will be reported as โ€œunknownโ€ #6135

    ๐Ÿ†• New debug handlers have been added to set block and mutex profiling. #6137

    0๏ธโƒฃ Vitessdriver: Allow overriding driver name. Default stays as "vitess" #6138

    โœ… VTCombo: add options to allow starting MySQL within VTCombo. Note that vttestserver provides similar functionality. #6265

    โœ… Disable foreign key checks during preflight schema change tests #4696

    ๐Ÿ›  Fix labels for opentsdb #6289

    ๐Ÿ‘ Allow empty shards when running ValidateSchemaKeyspace behind flag โ€˜-skip-no-masterโ€™ #6216

    Death by SIGPIPE on stdout/stderr can be avoided by setting the new flag โ€˜catch-sigpipeโ€™ #6297

    ๐Ÿ‘ Field query support in shard targeting #6132

    ๐Ÿ“š Documentation

    ๐Ÿ›  Fix links to local example and sample client in vitessdriver godoc #6254

    ๐Ÿ“š Flag documentation has been improved #6261

    ๐Ÿ“„ All flags are now documented in the program reference on the website.

    ๐Ÿ— Build Environment Changes

    ๐Ÿ‘• golangci-lint now runs as a pre-commit hook and CI check #6385 #6406

    โœ… CI test for region_sharding example #6275

    โœ‚ Removed staticcheck as a pre-commit hook because it is included in golangci-lint #6416

    โœ… Faster unit tests #6415

    โฌ†๏ธ Consul-api version upgraded to 1.8.0 #6358

    โฌ†๏ธ Upgrade pgzip to v1.2.4 #6189

    โฌ†๏ธ Upgrade log4j2 version to 2.13.3 #6403

    โž• Add docker_local target to Makefile and a docker/local/run.sh script #6393

    ๐Ÿง Build: Only check for k3s on Linux #6353

    Functionality Neutral Changes

    โž• Add context to MysqlDaemon schema functions. This lets us handle disconnected RPCs instead of the client hanging forever. #6243

    ๐Ÿ”จ Refactor table filtering logic #6242 #6259

    ๐Ÿ–จ VTGate: Replace Sprintf with string concatenation in normalizer, this reduces CPU usage on benchmarks #6127

    VTTablet: Reduce the number of packets sent from vttablet to MySQL #6130

    ๐Ÿ›  Fixed flakiness in External Connector Test: fixed race #6201

    ๐Ÿ›  Fixed flakiness in Message Test: test had not been updated after jitter functionality was added #6283

    ๐Ÿ›  Fix flakiness in TestConnectTimeout #6188

    ๐Ÿ›  Fix TLS test for Go 1.13+ #6185

    ๐Ÿ›  Fixed linter errors #6416 #6364 #6307

    โœ… More dollar-sign tests #6363

    VTExplain: use DiscoveryGateway (and old healthcheck) until migration can be completed #6248

    ๐Ÿ—„ Deprecated RPCs from 6.0 have been deleted #6380

    โšก๏ธ Update references to orchestrator repo to point to openark instead of github #6360

    ๐Ÿ—„ Terminology: deprecate or replace references to "slave" with โ€œreplicaโ€ except where referring to actual sql commands or fields returned from MySQL. #6428 #6392 #6379

    ๐Ÿ›  Fix log format - use Infof instead of Info #6413

    โœ… Cleanup temp files used in tests #6400

    โž• Add unit test for func stringMapToString #6280

    โœ… Check errors in etcd2topo unit test #6128

    โœ… Unit test for atomic.go #6120

    โœ… Unit test for ResolveIPv4Addrs #6230

    โœ… Enabled set statement tests #6167