Changelog History
Page 1
-
v8.0.0 Changes
October 27, 2020π This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.
Incompatible Changes
π The following PRs made changes to behavior that clients might be relying on. They should be reviewed carefully so that client code can be changed in concert with a Vitess release deployment.
- π Change error code from 1203 to 1153 for three specific error conditions. #6630
The three errors being changed:
When a GRPC message coming back to the vtgate exceeds the configured size limit (configured via grpc_max_message_size)
π§ If a write is attempting to make a change with a payload that is larger than the configured allowable size
If a scatter query is attempting to collect and process too many rows in memory within the vtgate to construct a response - 0οΈβ£ Zero auto-increment mode: This enables inserting a 0 value (and not just NULL) into an auto-increment column to generate sequence values. This matches the default behavior of MySQL. #6749
- 0οΈβ£ Turn off schema tracker by default #6712
π Bugs Fixed
VTGate / MySQL compatibility
- π Fix where clause in information schema with correct database name #6599
- π Fix DDL execution on reserved connection without in active transaction #6514
- Fix allow passed table_schema in information_schema queries to vttablet if the keyspace is not found #6653
- π Fix information schema for non-existent schema #6667
- π Fix reserved connection in autocommit mode on DML #6748
- π Fix dbname that needs escaping that we broke in fce4cfd #6794
- MySQL CLI, mysql --ssl does not work in MySQL 8.0 #5556
- OperationalError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server during query') #6208
- Fix support for
show tables where tables_in_db
to use keyspace name, not underlying db name #6446 - Common empty result set query on MySQL returns a row in Vitess #6663
- Operator precedence issue #6734
- Table names out-of-order between replica and primary #6738
- βͺ Close Idle reserved connections and Rollback Idle transactions #6552
- π Minor Vindex fixes: #6526
- VTgate does not report autocommit system variable and SERVER_STATUS_IN_TRANS flag correctly #5825
- 0οΈβ£ MariaDB JDBC Connector fails to connect to vtgate with default options #5851
- VTgate panics if there are no vttablets and vtctlds #6117
- β Remove shard session held in vtgate for reserved connection on connection failure #6522
- Session variable does not work as expected in 7.0 #6559
- Retry should not happen when in dedicated connection #6523
- Correctly report AUTOCOMMIT status in network packets #6551
- βͺ Close Idle reserved connections and Rollback Idle transactions #6552
- Reset Session for Reserved Connection Query Failure #6673
- Errors in multi-statement queries not fatal to statement sequence #6747 #6808
- π Lock wait timeout on DELETE from table using lookup index #6698
- Application couldn't connect to external db giving error tabletserver.go:1522] Code: INVALID_ARGUMENT syntax error at position 33 near ':vtg2' (CallerID: gnocchi) : Sql: "select convert(:vtg1, CHAR(:vtg2)) as anon_1 from dual", #6466
- The client got the wrong database name #6753
- π» DBAPIError exception wrapped from (pymysql.err.InternalError) (1105, u'vtgate: http://gnocchi-gnocchizone-vtgate:15000/: reserved connections are not supported on old gen gateway') #6577
- π Provide support for βSELECT INTO OUTFILE S3β #6811
- π Fix error around breaking of multi statements #6824
- π Support SHOW TABLE STATUS FROM db type queries in vtgate #6354
- π SHOW FULL FIELDS doesn't follow routing rules #6803
- VDiff/Tablet Picker: fix picker retry logic to handle canceled context #6954
- Healthcheck: Use isIncluded correctly to fix replica/rdonly routing bug #6922
Other
- VReplication tablet_picker should keep trying to find a tablet until context expires #6546
- β‘οΈ VReplication improve reverse workflow: update cells/tablet_types #6556
- VDiff: fix panic for tables with a unicode_loose_md5 vindex #6640
- π» Vtctld UI: Add mysqld port display back to vtctld web UI #6598 #6597
- π¨ Vtctld: Fix workflow CopyState fetching (and thus printing). #6657
- π Fix unsaved event regression #6710
- π Change tracker to use allprivs instead of dba #6720
- π Fixes long wait filter keyspace #6721
- s3 ListBackups doesnβt work from root directory #6732
- Vtctld: ExecuteFetchAsDba and ExecuteFetchAsAllPrivs do not escape db names. #6545
- Can't switch on restore_from_backup on existing shard with no tables #4896
- π¨ Refactor conn.go #6818
Functionality Added or Changed
VTGate / MySQL compatibility
- β Add support for DELETE IGNORE #6722
- Only take on simple dual queries in the vtgate #6666
- π Support mycli access to vtgate #4365
- Add support for unicode_loose_xxhash #6457
- Provide virtual information_schema #5394
- information_schema.tables_schema comparison with dynamic value no longer works #6827
- Mysqldump: Allow setting multiple variables in one statement #5401
- π Block lock function usage other than dual table #6470
- π Lock Session Support #6517
- π Make sure lookup vindexes are queryable inside transaction #6499
- Add unicode_loose_xxhash Vindex type #6549
- π Support vindex update or delete when destination is provided by client in dml query #6554
- π Make sure to handle EXPLAIN on tablets #6579
- Rewrite SHOW TABLES #6615
- π Make offset work in OLAP mode #6655
- Support sql_calc_found_rows with limit in sharded keyspace #6680
- π Lock shard session heartbeat #6683
- π Mysql -ssl tag is deprecated in the later versions of mysql #6682
- π Use statement support in olap mode #6692
- β Delete row using consistent lookup vindex in where clause #6700
- β Add vtexplain support for non-evenly sharded keyspaces #6705
- π Make sure dual queries set found_rows correctly #6718
- β Add if not exists support to database ddl in parsing #6724
- π Allow Update of same vindex using same vindex lookup on consistent lookup vindex #6736
- β‘οΈ Update Vindex only on changes #6737
- Add support for
SHOW (DATABASES|VITESS_SHARDS|VITESS_TABLETS) LIKE
#6750 - π 'CASCADE' support #6743
- Mask database name #6791
- π Make sure to backtick schema names properly #6550
- Transaction Limiter: fix quota leak if Begin() errors or if the tx killer runs #6731
- π Allow Load Data From S3 #6823
OLAP Functioanlity
- π Allow switching between OLAP and OLTP #6691
- π Use statement support in olap mode #6692
- π Fix error handling in olap mode #6940
π Set Statement Support
Set statement support is added in Vitess. There are some system variables which are disabled by default and can be enabled using flag
-enable_system_settings
on VTGate.These system variables are set on the backing MySQL instance, and will force the connection to be dedicated instead of part of the connection pool.- 0οΈβ£ Disabled passthrough system variables by default. #6859
- π Allow switching workload between OLAP and OLTP #4086 #6691
- π Support for multiple session set statement syntax #6508 #6495 #6494
- π Allow enumeration in set system variables #6493
- π Handle boolean settings better #6501
- π¨ Set statements refactored to use plan building #6487
- β Add more system settings #6486
- Evaluate system variables #6708
VReplication
- VReplication: _vt.vreplication source column VARBINARY->BLOB, allowing more tables to participate in a workflow #6421
- π VReplication and vstreamer performance and error metrics #6519, #6132
- VReplication: Find collations for char primary keys and cast comparison during catchup #6568
- Varchar primary keys with collations can cause vreplication to miss rows #6622
- VReplication: Timestamp w/ Heartbeat #6635
- π² VReplication: Improve few error/log messages #6669
- VReplication: handle SET statements in the binlog stream #6772
- π· VReplication: e2e Tests added to CI #6730
- π VDiff logs: additional logging around vdiff/vstreamer for observability #6684
- VStream: experimental POC of mysql protocol #6670
- π Materialize: Support rollups using a literal value as a column #6733
- Materialization: Ignore non-participating tables that are renamed/deleted #6778
- Materialize: add ability for maintaining accurate table-level aggregates #6767
- 0οΈβ£ Schema Tracker: Turn off schema tracker by default #6712 (breaking change)
- Workflow: Add List-All Command #6533
- π Workflow: List -> Show, and Expand Metadata #6544
- π¨ Refactor: Cleanup Noisy Workflow Logging #6744
- Schema tracker: Change tracker to use the allprivs user instead of dba #6720
- vtctl: Adding VExec and Workflow commands #6410
- vtctl Workflow cmd: don't expect all shards to have workflow streams #6576
- Reshard: Sort table definitions before comparing them #6765 #6738
- π¦ Expose a shard range calculator as a vtctl command GenerateShardRanges #6751 #6752
- VStream: Field event now has all column attributes #6525
- VStream events do not populate field "Flags" on FIELD events #5914
- π― RFC: VReplication based SplitClone #4604
- π Experimental: Mysql Protocol support for VStream #6675
- VReplication: Add metrics around vstreamer and vreplication metrics #6787
- π Fix issues with missing tables in vstreams #6789
- β Add validations and logs to fix common issues faced during MoveTables/Reshard #6814
- π Allow multiple blacklists for master #6816
VTtablet
- VTtablet throttling #6668
π https://vitess.io/docs/reference/features/tablet-throttler/ - TabletManager: publish displayState #6648
- TabletManager: call setReadOnly inside ChangeTabletType #6762
- TabletManager: change how SetReadOnly is called to avoid errors from externally managed tablets #6786
- TabletManager: call SetReadOnly inside ChangeTabletType #6804
- vttablet: Open and Close healthStreamer #6515
- vttablet: create database if not present #6490
- βͺ Restore: do not change tablet type to RESTORE if not actually performing a restore #6679
- βͺ Restore: checkNoDB should not require tables to be present #6695
- Heartbeat: Additional metrics/stats (lag histogram) #6528 #6634
- VTtablet throttler feature flag: -enable-lag-throttler #6815
- VTtablet two-phase commit design doc #6498
VTorc - Orchestrator
Following PRs are experimental version of Vitess-native Orchestrator 'vtorc' is ready for users to try out.
- Orchestrator: initial import of Orchestrator #6582
- Orchestrator: vitess mode #6613
- Orchestrator: more changes #6714
- β± Orchestrator: use contexts with timeout for remote operations #6780
- π Orchestrator: fixed orchestrator govet errors #6781
- Orchestrator: Add clusters_to_watch flag to. Defaults to all clusters. #6821
Point in Time Recovery
Other
- β Add realtime health stats to vtctld /api/keyspace/ks/tablets API #6569
- π» vtctld UI: Fix logic for displaying vindexes #6603
- π Online schema changes #6547
- π Table lifecycle management #6719, docs
- Online-DDL: migration uses low priority throttling #6837
- π¨ Emergency Reparent Refactor #6449
- Make emergency reparents more robust. #6206
- β Reparent test failures #6706
- β‘οΈ Update user-facing terminology in the vtctld2 UI #6481
- π€ GetSchema: Batch/parallel access to underlying mysqld for lower latency #6491
- Prometheus interface: vtgate_buffer_* metrics cardinality #6326
- π» Show more vindex details in vtctld /app/schema UI #6450
- β‘οΈ GRPC: update enforcement policy on server to match the one from the client #6629
Examples / Tutorials
- region_sharding: working resharding example #6565
- π³ vtcompose/docker-compose: fix InitShardMaster #6584
- π³ [docker] fix tablet hostname #6509
π Documentation
- Replace more uses of slave with replica/replication #6520
- β‘οΈ Update some option docstrings for accuracy. #6590
- Minor addition to Materialize help text. #6627
- β Remove auto-generation comment #6728
- β add skip flag that can skip comparing source & destination schema when run splitdiff #6477
- π Fix and clarify some help options #6817
π Build Environment Changes
- β Address #6631 by reducing optimization level #6632
- β Add front-end build tooling + fix build error #6473
- Replace gogo proto with golang #6571
- π Improve make proto #6580
- β Remove chromium hard-coding #6636
- π Docker - allow BUILD_NUMBER variable override #6628
- π³ Docker/lite: Fix package URLs for ubi images. #6704
- β‘οΈ Helm & docker/k8s update #6723
- π Fix build errors in ./go/vt/vtgate for go1.15 #6800
- π Fix build error on vcursor_impl tests on newer versions of go #6799
- π [java] bump java version to 8.0.0-SNAPSHOT for next release #6478
- π Helm/docker: fix Docker builds + tag 7.0.2 #6773
- π Docker: Upgrade to Debian Buster (release-8.0) #6888
- π³ Docker: Pin mysql 5.7 and 8.0 versions for xtrabackup compatibility
- π Docker: Revert "Docker - upgrade to Debian Buster (release-8.0)" #6929
- Zookeper: Download zookeeper 3.4.14 from archive site #6867
Functionality Neutral Changes
- π Routing cleanup: remove routeOptions #6531
- β PITR: testcase #6594
- Replaced Error with a warning in case parsing of VT_SPAN_CONTEXT fails #6766
- π¨ Reparent tests refactoring: setup/teardown for each test, reduce cut/paste, improve readability #6726
- β Decider: endtoend test infrastructure + tests #6770
- β Orchestrator: add more test cases #6801
- β Orchestrator: don't set DbName for tests that don't need it. #6812
- β Add unit test case to improve test coverage for go/sqltypes/named_result.go #6672
- β End to end: deflake sharding tests #6715
- π Workflow test: fix flaky test. remove obsolete code #6694
- β Add unit test for func Proto3ValuesEqual. #6649
- β Ensure tests for discoverygateway #6536
- π² Convert syslog events to normal log events #6511
- β Add diagnostic logging to healthcheck. #6512
- Healthcheck should receive healthcheck updates from all tablets in cells_to_watch #6852
- Named columns and convenience SQL query interface #6543
- β Check http response code in vtctld API tests #6570
- π Reverting package-lock.json to the one pre-PR #6603 #6611
- β Removed sqlparser.preview to set logstats stmtType and use plan.type #6637
- AST struct name rewording #6642
- π Fix: Remove SetMaster Query Expectation from ERS Test #6617
- π Fix path for vttablet query log in end to end test #6656
- π Fixes error log #6496
- β Add diagnostic logging to healthcheck. #6535
- Try to install apt keys from list of different keyservers #6674
- Trivial copy-pasta comment fixup #6592
- β Remove long-unused memcache and cacheservice #6596
- π Fix MoveTables docstring; was not valid JSON. #6606
- β Tablet gateway: unit tests #6608
- Tablet gateway doesn't see all tablets in cells from cells_to_watch #6846
- β Add Sleep to ERS Unit Test #6610
- Strings to enum #6729
- β Remove syslog dependency #3563
- Operator precedence must take associativity into consideration #6758
- β Add optional ignore to delete statement #6802
- π° Invalid trace payload causes vitess to fail the request and return an error #6759
- π² Vttablet: minor logging fix #6564
- π² Vttablet: add more logging to checkMastership #6618
- π Change error code from 1203 to 1153 for three specific error conditions. #6630
-
v8.0.0-test
October 26, 2020 -
v8.0.0-rc1 Changes
October 07, 2020π This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.
Incompatible Changes
π The following PRs made changes to behavior that clients might be relying on. They should be reviewed carefully so that client code can be changed in concert with a Vitess release deployment.
- π Change error code from 1203 to 1153 for three specific error conditions. #6630
The three errors being changed:
When a GRPC message coming back to the vtgate exceeds the configured size limit (configured via grpc_max_message_size)
π§ If a write is attempting to make a change with a payload that is larger than the configured allowable size
If a scatter query is attempting to collect and process too many rows in memory within the vtgate to construct a response - 0οΈβ£ Zero auto-increment mode: This enables inserting a 0 value (and not just NULL) into an auto-increment column to generate sequence values. This matches the default behavior of MySQL. #6749
- 0οΈβ£ Turn off schema tracker by default #6712
π Bugs Fixed
VTGate / MySQL compatibility
- π Fix where clause in information schema with correct database name #6599
- π Fix DDL execution on reserved connection without in active transaction #6514
- Fix allow passed table_schema in information_schema queries to vttablet if the keyspace is not found #6653
- π Fix information schema for non-existent schema #6667
- π Fix reserved connection in autocommit mode on DML #6748
- π Fix dbname that needs escaping that we broke in fce4cfd #6794
- MySQL CLI, mysql --ssl does not work in MySQL 8.0 #5556
- OperationalError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server during query') #6208
- Fix support for
show tables where tables_in_db
to use keyspace name, not underlying db name #6446 - Common empty result set query on MySQL returns a row in Vitess #6663
- Operator precedence issue #6734
- Table names out-of-order between replica and primary #6738
- βͺ Close Idle reserved connections and Rollback Idle transactions #6552
- π Minor Vindex fixes: #6526
- VTgate does not report autocommit system variable and SERVER_STATUS_IN_TRANS flag correctly #5825
- 0οΈβ£ MariaDB JDBC Connector fails to connect to vtgate with default options #5851
- VTgate panics if there are no vttablets and vtctlds #6117
- β Remove shard session held in vtgate for reserved connection on connection failure #6522
- Session variable does not work as expected in 7.0 #6559
- Retry should not happen when in dedicated connection #6523
- Correctly report AUTOCOMMIT status in network packets #6551
- βͺ Close Idle reserved connections and Rollback Idle transactions #6552
- Reset Session for Reserved Connection Query Failure #6673
- Errors in multi-statement queries not fatal to statement sequence #6747 #6808
- π Lock wait timeout on DELETE from table using lookup index #6698
- Application couldn't connect to external db giving error tabletserver.go:1522] Code: INVALID_ARGUMENT syntax error at position 33 near ':vtg2' (CallerID: gnocchi) : Sql: "select convert(:vtg1, CHAR(:vtg2)) as anon_1 from dual", #6466
- The client got the wrong database name #6753
- π» DBAPIError exception wrapped from (pymysql.err.InternalError) (1105, u'vtgate: http://gnocchi-gnocchizone-vtgate:15000/: reserved connections are not supported on old gen gateway') #6577
- π Provide support for βSELECT INTO OUTFILE S3β #6811
Other
- VReplication tablet_picker should keep trying to find a tablet until context expires #6546
- β‘οΈ VReplication improve reverse workflow: update cells/tablet_types #6556
- VDiff: fix panic for tables with a unicode_loose_md5 vindex #6640
- π» Vtctld UI: Add mysqld port display back to vtctld web UI #6598 #6597
- π¨ Vtctld: Fix workflow CopyState fetching (and thus printing). #6657
- π Fix unsaved event regression #6710
- π Change tracker to use allprivs instead of dba #6720
- π Fixes long wait filter keyspace #6721
- s3 ListBackups doesnβt work from root directory #6732
- Vtctld: ExecuteFetchAsDba and ExecuteFetchAsAllPrivs do not escape db names. #6545
- Can't switch on restore_from_backup on existing shard with no tables #4896
- π¨ Refactor conn.go #6818
Functionality Added or Changed
VTGate / MySQL compatibility
π New functionality to set workload = 'olap'; can not repeat, but set workload = 'oltp' can; #4086
β Add support for DELETE IGNORE #6722
Only take on simple dual queries in the vtgate #6666
π Support mycli access to vtgate #4365
Add support for unicode_loose_xxhash #6457
Provide virtual information_schema #5394
Mysqldump: Allow setting multiple variables in one statement #5401
SET planning #6487
π Allow enumeration in set system variables #6493
π Allow set to have scope for each setting #6494
π HandleSet in executor to handle multiple settings in single set statement #6495
0οΈβ£ Passthrough system settings flag: enabled by default #6586
π Handle boolean settings better #6501
β Add more system settings #6486
π Block lock function usage other than dual table #6470
π Lock Session Support #6517
π Make sure lookup vindexes are queryable inside transaction #6499
Add unicode_loose_xxhash Vindex type #6549
π Support vindex update or delete when destination is provided by client in dml query #6554
π Make sure to handle EXPLAIN on tablets #6579
Rewrite SHOW TABLES #6615
π Make offset work in OLAP mode #6655
Support sql_calc_found_rows with limit in sharded keyspace #6680
π Lock shard session heartbeat #6683
π Mysql -ssl tag is deprecated in the later versions of mysql #6682
π Allow switching between OLAP and OLTP #6691
π Use statement support in olap mode #6692
β Delete row using consistent lookup vindex in where clause #6700
β Add vtexplain support for non-evenly sharded keyspaces #6705
Evaluate system variables #6708
π Make sure dual queries set found_rows correctly #6718
β Add if not exists support to database ddl in parsing #6724
π Allow Update of same vindex using same vindex lookup on consistent lookup vindex #6736
β‘οΈ Update Vindex only on changes #6737
Add support for
SHOW (DATABASES|VITESS_SHARDS|VITESS_TABLETS) LIKE
#6750π Support multiple session set statement syntax #6508
π 'CASCADE' support #6743
Mask database name #6791
π Make sure to backtick schema names properly #6550
Transaction Limiter: fix quota leak if Begin() errors or if the tx killer runs #6731
OLAP Functioanlity
System Variables
- π Allow enumeration in set system variables #6493
- π Allow set to have scope for each setting #6494
- π HandleSet in executor to handle multiple settings in single set statement #6495
VReplication
- VReplication: _vt.vreplication source column VARBINARY->BLOB, allowing more tables to participate in a workflow #6421
- π VReplication and vstreamer performance and error metrics #6519, #6132
- VReplication: Find collations for char primary keys and cast comparison during catchup #6568
- Varchar primary keys with collations can cause vreplication to miss rows #6622
- VReplication: Timestamp w/ Heartbeat #6635
- π² VReplication: Improve few error/log messages #6669
- VReplication: handle SET statements in the binlog stream #6772
- π· VReplication: e2e Tests added to CI #6730
- π VDiff logs: additional logging around vdiff/vstreamer for observability #6684
- VStream: experimental POC of mysql protocol #6670
- π Materialize: Support rollups using a literal value as a column #6733
- Materialization: Ignore non-participating tables that are renamed/deleted #6778
- Materialize: add ability for maintaining accurate table-level aggregates #6767
- 0οΈβ£ Schema Tracker: Turn off schema tracker by default #6712 (breaking change)
- Workflow: Add List-All Command #6533
- π Workflow: List -> Show, and Expand Metadata #6544
- π¨ Refactor: Cleanup Noisy Workflow Logging #6744
- Schema tracker: Change tracker to use the allprivs user instead of dba #6720
- vtctl: Adding VExec and Workflow commands #6410
- vtctl Workflow cmd: don't expect all shards to have workflow streams #6576
- Reshard: Sort table definitions before comparing them #6765 #6738
- π¦ Expose a shard range calculator as a vtctl command GenerateShardRanges #6751 #6752
- VStream: Field event now has all column attributes #6525
- VStream events do not populate field "Flags" on FIELD events #5914
- π― RFC: VReplication based SplitClone #4604
- π Experimental: Mysql Protocol support for VStream #6675
- VReplication: Add metrics around vstreamer and vreplication metrics #6787
- π Fix issues with missing tables in vstreams #6789
- β Add validations and logs to fix common issues faced during MoveTables/Reshard #6814
VTtablet
- VTtablet throttling #6668
- TabletManager: publish displayState #6648
- TabletManager: call setReadOnly inside ChangeTabletType #6762
- TabletManager: change how SetReadOnly is called to avoid errors from externally managed tablets #6786
- TabletManager: call SetReadOnly inside ChangeTabletType #6804
- vttablet: Open and Close healthStreamer #6515
- vttablet: create database if not present #6490
- βͺ Restore: do not change tablet type to RESTORE if not actually performing a restore #6679
- βͺ Restore: checkNoDB should not require tables to be present #6695
- Heartbeat: Additional metrics/stats (lag histogram) #6528 #6634
- VTtablet throttler feature flag: -enable-lag-throttler #6815
VTorc - Orchestrator
- Orchestrator: initial import of Orchestrator #6582
- Orchestrator: vitess mode #6613
- Orchestrator: more changes #6714
- β± Orchestrator: use contexts with timeout for remote operations #6780
- π Orchestrator: fixed orchestrator govet errors #6781
- Orchestrator: Add clusters_to_watch flag to. Defaults to all clusters. #6821
Point in Time Recovery
Other
- β Add realtime health stats to vtctld /api/keyspace/ks/tablets API #6569
- π» vtctld UI: Fix logic for displaying vindexes #6603
- Online schema changes #6547
- Table lifecycle management #6719, docs: https://vitess.io/docs/reference/features/table-lifecycle/
- π¨ Emergency Reparent Refactor #6449
- Make emergency reparents more robust. #6206
- β‘οΈ Update user-facing terminology in the vtctld2 UI #6481
- π€ GetSchema: Batch/parallel access to underlying mysqld for lower latency #6491
- Prometheus interface: vtgate_buffer_* metrics cardinality #6326
- π» Show more vindex details in vtctld /app/schema UI #6450
- β‘οΈ GRPC: update enforcement policy on server to match the one from the client #6629
Examples / Tutorials
- region_sharding: working resharding example #6565
- π³ vtcompose/docker-compose: fix InitShardMaster #6584
- π³ [docker] fix tablet hostname #6509
π Documentation
- Replace more uses of slave with replica/replication #6520
- β‘οΈ Update some option docstrings for accuracy. #6590
- Minor addition to Materialize help text. #6627
- β Remove auto-generation comment #6728
- β Add skip flag that can skip comparing source & destination schema when run splitdiff #6477
- π Fix and clarify some help options #6817
π Build Environment Changes
- β Address #6631 by reducing optimization level #6632
- β Add front-end build tooling + fix build error #6473
- Replace gogo proto with golang #6571
- π Improve make proto #6580
- β Remove chromium hard-coding #6636
- π Docker - allow BUILD_NUMBER variable override #6628
- π³ Docker/lite: Fix package URLs for ubi images. #6704
- β‘οΈ Helm & docker/k8s update #6723
- π Fix build errors in ./go/vt/vtgate for go1.15 #6800
- π Fix build error on vcursor_impl tests on newer versions of go #6799
- π [java] bump java version to 8.0.0-SNAPSHOT for next release #6478
- π Helm/docker: fix Docker builds + tag 7.0.2 #6773
Functionality Neutral Changes
- π Routing cleanup: remove routeOptions #6531
- β PITR: testcase #6594
- Replaced Error with a warning in case parsing of VT_SPAN_CONTEXT fails #6766
- π¨ Reparent tests refactoring: setup/teardown for each test, reduce cut/paste, improve readability #6726
- β Decider: endtoend test infrastructure + tests #6770
- β Orchestrator: add more test cases #6801
- β Orchestrator: don't set DbName for tests that don't need it. #6812
- β Add unit test case to improve test coverage for go/sqltypes/named_result.go #6672
- β End to end: deflake sharding tests #6715
- π Workflow test: fix flaky test. remove obsolete code #6694
- β Add unit test for func Proto3ValuesEqual. #6649
- β Ensure tests for discoverygateway #6536
- π² Convert syslog events to normal log events #6511
- β Add diagnostic logging to healthcheck. #6512
- Named columns and convenience SQL query interface #6543
- β Check http response code in vtctld API tests #6570
- π Reverting package-lock.json to the one pre-PR #6603 #6611
- β Removed sqlparser.preview to set logstats stmtType and use plan.type #6637
- AST struct name rewording #6642
- π Fix: Remove SetMaster Query Expectation from ERS Test #6617
- π Fix path for vttablet query log in end to end test #6656
- π Fixes error log #6496
- β Add diagnostic logging to healthcheck. #6535
- Try to install apt keys from list of different keyservers #6674
- Trivial copy-pasta comment fixup #6592
- β Remove long-unused memcache and cacheservice #6596
- π Fix MoveTables docstring; was not valid JSON. #6606
- β Tablet gateway: unit tests #6608
- β Add Sleep to ERS Unit Test #6610
- Strings to enum #6729
- β Remove syslog dependency #3563
- Operator precedence must take associativity into consideration #6758
- β Add optional ignore to delete statement #6802
- π° Invalid trace payload causes vitess to fail the request and return an error #6759
- π² Vttablet: minor logging fix #6564
- π² Vttablet: add more logging to checkMastership #6618
- π Change error code from 1203 to 1153 for three specific error conditions. #6630
-
v7.0.3 Changes
October 22, 2020π This update fixes several regressions that were deemed significant enough to be backported to the release branch.
π Bugs Fixed
- Vtgate : Should receive healthcheck updates from all tablets in cells_to_watch #6857
- Vtgate : Fix error around breaking of multistatements #6824
- Vtgate : Fix bug in REPLICA/RDONLY routing #6904
- Vttablet : checkNoDB should not require tables to be present #6788
Known Issues
π This release includes MySQL 5.7.32 and Percona Xtrabackup 2.4.20 which have a known incompatibility. Percona is working on a resolution.
The issue will manifest if a user using 5.7.32: * Creates a new compressed table. * Runs any ALTER TABLE that uses the algorithm copy (table rebuild) on a compressed table.
π Build Changes
- π Build : Download zookeeper 3.4.14 from archive site #6868
-
v7.0.2 Changes
September 22, 2020π This update fixes several regressions that were deemed significant enough to be backported to the release branch.
π Bugs Fixed
- vtgate : Operator precedence must take associativity into consideration #6764
- vtgate: Fix reserved connection in autocommit mode on DML #6748
- VDiff: fix panic for tables with a unicode_loose_md5 vindex #6745
- 0οΈβ£ vttablet: Turn off schema tracker by default #6746
- βͺ restore: do not change tablet type to RESTORE if not actually performing a restore #6687
- vttablet : Tablet manager init should publish displayState #6686
- π² vttablet: Tablet manager add more logging to checkMastership #6685
- vttablet : Fix replmanager deadlock #6634
-
v7.0.1 Changes
August 25, 2020π This update fixes several regressions that were deemed significant enough to be backported to the release branch.
0οΈβ£ In addition, the experimental feature to set system variables is now behind a flag that disables it by default.π§ Configuration Changes
- Vtgate: added flag enable_system_settings to enable passthrough system settings, defaults to false #6562
π Bugs Fixed
- Vreplication: Materialize should only get schema from source tablets if target is missing tables #6623
- π» Vtctld UI: mysql port is not displayed in tablet list #6597 #6600
- Vtgate: transaction killer was closing reserved connection after the specified time. Only idle connections should be closed. Idle transactions should be rolled back. #6583
- π³ Vtcompose: docker-compose example was broken by tabletmanager changes #6587
- Vreplication: tablet_picker should respect canceled context. Also, keep trying to find a tablet until context expires. #6567
- β‘οΈ Vtgate: vindex update/delete by destination #6541 #6572
- Vtgate: some usages of EXPLAIN were broken #6574 #6581
- Vtgate: handle multiple settings in single SET statement #6504
- Vtgate: cleanup shard session for reserved connection on failure #6563
- Vtgate: if executing DDL on reserved connection, start transaction if needed #6555
- Vtgate: make sure lookup vindexes use same transaction if one is open #6505
- π² Healthcheck: add diagnostic logging, do not add tablet if it is shutting down #6512
-
v7.0.0 Changes
July 28, 2020Incompatible Changes
π This release includes the following changes which may result in incompatibilities when upgrading from a previous release. It is important that Vitess components are upgraded in the recommended order. This will change in the next release as documented in VEP-3.
π VTGate: The default gateway (-gateway_implementation flag value) is now "tabletgateway". The old default was βdiscoverygatewayβ. To use the old gateway, specify the flag in the command line. Anyone already using the flag will see no change; to use the new gateway, remove the -gateway_implementation flag. The new gateway is simpler and uses a new healthcheck module with no loss of functionality. #6155
β VTGate: VTGate incorrectly returned Warning Code field name as Type. This has now been fixed. Code or scripts that depend on the incorrect behavior might break. #6437
π VReplication: The "-cell" parameter to MoveTables has been renamed to β-cellsβ, since it now accepts multiple comma-separated cells or cellsAliases. The parameter specifies which cells can provide tablets as the source for vreplication streams. #6456
VTGate: The possibility of sending OpenTracing trace spans through query comments has been changed. The old way of doing it was severely broken, so our assumption is that no-one has successfully been using this feature. Still, this is an API change, and we are reporting it as a breaking change. #6463
π¨ VTTablet has been refactored to be generally more responsive to state changes and its environment: #6131 #6139 #6241 #6263 #6311 #6345 #6348 #6384 #6396 #6432 #6461
Some initialization parameters are now mandatory: init_keyspace, init_shard and init_tablet_type. The flow where you could issue a vtctlclient InitTablet and then invoke vttablet without those init parameters is no longer supported.
β‘οΈ Consequently vtctl's InitTablet and UpdateTabletAddrs are deprecated.
demote_master_type is deprecated: Now that init_tablet_type is mandatory, that value can act as the demote_master_type. There is no need for a separate flag.
β‘οΈ VTTablet will publish the discovered MySQL port on startup, and will not update it during the life of the process. There is no known use case where a MySQL port is expected to change. So, there is no need to keep polling MySQL for its port.
Decouple enable_replication_watcher from disable_active_reparents. Currently, vttablet attempts to fix replication only if enable_replication_watcher is true and disable_active_reparents is false. This behavior has changed to depend only on the disable_active_reparents flag.
Transitioning from a serving type to SPARE used to go through a brief lameduck period specified by the serving_state_grace_period (default 0). This has been removed. The transition will now happen immediately even if a grace period is specified.
π It was possible to enable health reporting and heartbeat at the same time. Performing the two at the same time is not useful. As of this release, enabling heartbeat automatically disables the polling mode of checking for replica health.
π The ACL rules used to be reported on the vttablet status page. The design was using an incorrect framework. This has been removed.
π Build: make targets "build_web" and βembed_staticβ have been removed. Instead there are new targets βweb_bootstrapβ, βweb_buildβ and βweb_startβ #6473
π Deprecations
π The following vtctl commands have been deprecated. Deprecated commands will continue to work until the next release (8.0). #6428 #6345 #6461
StartSlave -> Replaced with StartReplication
StopSlave -> Replaced with StopReplication
π ChangeSlaveType -> Replaced with ChangeTabletType
InitTablet
β‘οΈ UpdateTabletAddrs
Various vtctl commands had a flag called "-wait_slave_timeout". It has been replaced with β-wait_replicas_timeoutβ. Scripts using the old flag will continue to work until the next major release (8.0). #6428
π Helm charts: The current helm charts have not been keeping up with the forward progress in vitess, and there has been no maintainer for it. Additionally, the operator has pulled ahead of what the helm charts could do. The operator can also be functionally more versatile than the helm charts because custom code can be written to manage complex workflows. To maintain focus and velocity in the project, we are deprecating the helm charts. #6439
π Consul: The usage of Consul has been problematic for vitess users. The main difficulty comes from the fact that it's non-trivial for a multi-cell deployment. Due to this, we want to discourage the use of consul as a topo server, and are deprecating support for it. #6440
π Bugs Fixed
VTGate / MySQL compatibility
π Handle non-keyword tokens FORMAT, TREE and VITESS. (This was broken by #6110) #6445
When using user-defined variables that have not been initialized first, a non-fatal panic occurs and the connection is dropped. #6294 #6308
Setting system variable panics vtgate. #6162 #6163
regression: USE does not work from MySQL client. #6153 #6157
π USE does not return an error if a keyspace doesn't exist. #3060 #6157
Dotted table names are not escaped properly in lookup vindex flow. #6142 #6145
Produce correct error code (1105 instead of 1062) when inserting duplicates into consistent_lookup vindex. #6144
Rewriting expressions fail when used as function parameters. #6281
β‘οΈ Rollback after error clobbers updates. #6285
Queries should not be wrapped into transactions if autocommit=1 is set. #6264
Prepared statements with bind variables in column specification produce βmissing bind varβ error. #6287 #6298
π Prepared statement queries returning NULL results are not parsed. #6240 #6249
0οΈβ£ DEFAULT function not working with status column. #6221 #6224
VTGate panic kills connection when selecting database() when using OLAP workload. #6200 #6205
Wrong column value can be inserted due to parameterization variable confusion. #6266 #6270
π Fix owned table name formatting and duplicate error code in consistent lookup. #6145
π Support for case-insensitive user defined variables. #6225
π Fix performance regression on VTGate. #6203
Return NULL for non-existent user-defined variables. #6308
MySQL connection closing should not inflate query counts. #6226
π show vitess_shards intermittently returns empty list. #5038 #5189
Protect insert lookup queries from bind variable name clashes #6328
CAST/CONVERT should work correctly #6304
π Allow charset introducers for vindex columns #6301
π Fix panic from EXPLAIN #6204
π Handle special comments that start with ! #6186
π Fix panic with uppercase types #6140
π Allow β$β in identifiers #6136
Other
β± VTTablet: Etcd query timeout because auto-committable transactions are not properly rolled back. #6258 #6299
π VTTablet: Fixed race in schema engine during notification of schema changes. #6268
π VTTablet: Reloading ACL config was throwing misleading error messages.his has now been fixed and the error messages improved. #6309
π VTTablet: Panic during backup caused an error in the pgzip library has been fixed by contributing a fix to pgzip and upgrading to a new release that includes the fix. #6189
VTTablet: Improvement of error handling by retrying while uploading to S3. #6182 #6395 #6405
π VTTablet: builtinbackupengine was uploading a manifest despite errors from S3. Error collection during backup has been fixed to address this. #6349 #6350
VTTablet: Fix dba connection leak in LockTables. #6424
VTTablet: Fix connection pool leak in schema engine. #6426
VTTablet: Fix regression where a heartbeat interval flag setting of 1s would have resulted in an internal value of 1000s #6356 #6357
VTTablet: Fix panic from binlog_streamer when β-watch_replication_streamβ is set., #6253
VTTablet: Add context to MysqlDaemon schema functions. This lets us handle disconnected RPCs instead of the client hanging forever. #6243
VTTablet: fix panic from txlogz #6238
π VTTablet: Add back support for millisecond values for flags that was dropped when yaml config was introduced #6250
π» Vtctld UI: Change all external links in vitess to use https #6170 #6169
Ensure SrvVSchema gets rebuilt for all cells #6276
π Fix MariadbGTIDSet multi-domain support #6184
mysqlctl: Fix connection leak during killConnection() #6245
π Fix release script to install examples in the correct directory #6159
examples/operator: fix port-forward command #6418
VStream client: should send current position events immediately #6391 #6397
Functionality Added or Changed
VTGate/ MySQL compatibility
Transactions on read-only replicas are now supported. Start a transaction by first choosing the desired tablet_type ("use keyspace@replica"), then issuing a βBEGINβ statement. Internally, this will issue a βSTART TRANSACTION READ ONLYβ on the underlying database. This is only available if using the new gateway_implementation of tabletgateway (which is now the default). #6166 #6244
π Create better plans for NULL comparisons. Instead of using a SelectScatter, now vtgate will produce empty results when the predicate is a comparison against literal NULLs. #6133 #6425
π Support IN, NOT IN and equality (=) with null values #6152 #6146
β Add EXPLAIN support for vtgate plans. View the produced vtgate query plan by using explain format=vitess #6110
π Allow users to perform savepoint related constructs. #6355 #6404 #6412 #6414
π Added support for user-level advisory lock functions like get_lock and release_lock. This is considered experimental and will be improved in future releases based on user feedback. #6367 #6370 #6470
π Allow users to change system variable settings at session level using set statements. This is a major change that was merged later in the development cycle. It is considered experimental and there may be edge cases. #6107 #6459 #6472 #6488
"ignore_nulls" option to lookup vindexes #6147 #6222: There are situations where the "from" columns of a lookup vindex can be null. Such columns cannot be inserted in the lookup due to the uniqueness constraints of a lookup. The new βignore_nullsβ option of the lookup vindex allows you to accommodate null values in rows. Such rows can still be found through other means, like the primary vindex.
β Add support for row_count() function. #6174 #6179
π XOR Operator is now supported. #6369 #6371
Allow 'binary' for character_set_results. It is equivalent to null. #6237
π Improve logging by adding keyspace, table, and tablet_type. #6282
π° VTGate now takes in optional flags for βmax_payload_sizeβ and βwarn_payload_sizeβ. A payload size above the warning threshold will only produce a warning. A payload size that exceeds βmax_payload_sizeβ will result in an error. The default behavior is not to check payload size, which is backwards-compatible. #6143 #6375
A new comment directive (β/*vt+ IGNORE_MAX_PAYLOAD_SIZE=1 */) has been added to provide the ability to override the payload size checking per query. #6143
A new comment directive (β/*vt+ IGNORE_MAX_MEMORY_ROWS=1 */) has been added to provide the ability to ignore the "-max_memory_rows" setting per query. #6430
Support βselect * from information_schema.processlistβ. #6310
π Support batch lookup queries for integer and binary types. Other types will continue to require one query per row. #6420
π Support system settings at session level. #6149
Prepare for future features by adding reserved connections. This is the ability for a connected client to have a dedicated connection instead of using the connection pool. #6303 #6313 #6402 #6441 #6451
π Allow charset introducers with vindex column values. #6236 #6314
π Support Union All between different shards. #6351
Exclude KeyspaceId(s) from plan cache key.This improves the cache hit rate. #6233
Reload TLS config on SIGHUP. #6215
β¬οΈ Drop vindex DDL. #6151
show vitess_tablets now reports master_term_start_time. We ensure that this is reported correctly after reparents. #6135 #6292 #6293
π Allow parenthesized selects #6342
π Support DO statements #6239
π Support calling database() inside other functions #6291
π Support IN, NOT IN and equality (=) with null values #6152 #6146
VReplication
VStream Copy extends the VStream API to allow streaming of the contents of databases or tables. Earlier you had to specify a position to start streaming events from. #6277
0οΈβ£ VReplication workflows now accept "-cells" as an option. This can be a comma-separated list of cells or cellsAliases. The source tablet for vreplication will be chosen randomly among the tablets that belong to the specified cells/cellsAliases. If the option is not provided, we default to the cell that is local to the vreplication target. #6442, #6456 #6465
Schema Versioning. Tablets now have a schema tracker that can be enabled to store schema versions as DDLs are applied. The _vt.schema_version table contains the schema with the corresponding ddl and gtid position when the ddl was applied. The tracked schemas are used by the new Historian service to provide vstreams with the correct schema while streaming old events. #6300 #6164 #6278 #6399 #6411 #6435 #6448
π Support for specifying multiple external sources to facilitate migration into vitess.
π Foreign Keys are now supported on the target database during vreplication. In the copy phase we disable FK checks and re-enable them once the replication phase starts. #6284
Materialize: added a drop_constraint option during table copy to strip constraints while creating target tables #6422
π Materialize: optimizations while creating target tables resulting in a significant improvement in performance when moving a large number of tables from an external MySQL instance #6207, #6251
β¬οΈ DropSources: Added β-rename_tablesβ option to rename the source tables rather than dropping them entirely. #6383
πΎ Savepoints: Vreplication streams are not able to handle savepoint events in the binlog. #6438
β Add keyspace, shard and workflow labels to VReplicationSecondsBehindMaster metrics #6279
π» Correct command line help for MigrateServedTypes, MigrateServedFrom and SwitchWrites #6187
VExec and Workflow commands have been added to vtctld as a layer above VReplicationExec. They allow listing/stopping/starting/deleting workflows and ability to run custom queries on the _vt.vreplication table for all shards in a workflow. #6410
Point-in-time Recovery
π In the 4.0 release we introduced functionality to recover a keyspace into a snapshot based on a specified time. That feature only allowed you to restore a backup, but did not have the ability to automatically replay binlogs up to the specified time. The feature has now been completed so that true point-in-time recovery is possible. This is currently only supported for MySQL compatible installations, not MariaDB. #4857 #6267 #6408
Healthcheck
π VTGate can now use a new healthcheck that is logically simpler while providing the same functionality. This is the default. The older healthcheck has been deprecated but is still available to use if desired by setting "-gateway_implementation=discoverygateway" #5750 #6155
VTCombo has been migrated to use the new healthcheck #6302
VTExplain has been migrated to use new healthcheck #6460
MiniVitess
π³ A docker container that can be used to bring up a self-contained vitess cluster, which can automatically resolve and self-configure to meet an external MySQL cluster. While this might sound very similar to VTCombo and vttestserver, it is not quite the same.
π³ MiniVitess spins up in docker. Given a MySQL server hostname, schema name & credentials, it runs an automated topology analysis via orchestrator, identifying the cluster members. It proceeds to bring up distinct tablet for each server and opens up vtgate connections. #6377
Examples / Tutorials
Vitess operator example #6154
Region sharding example #6209 #6275
π Change local example to use new command DropSources #6178
β Are-you-alive example now makes it easier to test multiple endpoints #6202
π³ Local docker tutorial added to website
π Simplified helm chart for examples (however, helm is now deprecated) #6168
Other
β‘οΈ vtctlclient ListAllTablets now shows the MasterTermStartTime. This is shown as "" for read-only tablet types. In case of an old master that has not yet updated the topo record, TabletType will be reported as βunknownβ #6135
π New debug handlers have been added to enable/disable golang block and mutex profiling. #6137
0οΈβ£ Vitessdriver: Allow overriding driver name. Default stays as "vitess" #6138
β VTCombo: add options to allow starting MySQL within VTCombo. Note that vttestserver provides similar functionality. #6265
β Disable foreign key checks during preflight schema change tests #4696
π Fix labels for opentsdb #6289
π Allow empty shards when running ValidateSchemaKeyspace behind flag β-skip-no-masterβ #6216
Death by SIGPIPE on stdout/stderr can be avoided by setting the new flag βcatch-sigpipeβ #6297
π Prepared Statement support added when using special vitess construct of use
<keyspace>:<shard>
by adding GetField support in engine plan execute #6132GTIDSet Union #6180
β Add relay log position and file based positions to ReplicationStatus #6217
Function that can find errant GTIDs #6296
VTTablet: Add StopReplicationMode to StopReplicationAndGetStatus RPC #6335
VTTablet: DemoteMaster RPC returns full status instead of string representation of current position #6365
VTTablet: WaitForPosition RPC can now accept either file-based or GTID-based positions #6374
π Documentation
π Fix links to local example and sample client in vitessdriver godoc #6254
π Flag documentation has been improved #6261
π All flags are now documented in the program reference on the website.
π Build Environment Changes
π golangci-lint now runs as a pre-commit hook and CI check #6385 #6406
β CI test for region_sharding example #6275
β Removed staticcheck as a pre-commit hook because it is included in golangci-lint #6416
β Faster unit tests #6415
β¬οΈ Consul-api version upgraded to 1.8.0 #6358
β¬οΈ Upgrade pgzip to v1.2.4 #6189
β¬οΈ Upgrade log4j2 version to 2.13.3 #6403
β¬οΈ Upgrade netty version to 4.1.48.Final #6317
β¬οΈ Upgrade tcnative version to 2.0.30.Final #6317
β Add docker_local target to Makefile and a docker/local/run.sh script #6393
π§ Only check for k3s on Linux #6353
π UI build tooling was broken, it has now been fixed. #6473
Functionality Neutral Changes
π¨ Refactor table filtering logic #6242 #6259
π¨ VTGate: Replace Sprintf with string concatenation in normalizer, this reduces CPU usage on benchmarks #6127
VTTablet: Reduce the number of packets sent from vttablet to MySQL #6130
π Fixed flakiness in External Connector Test: fixed race #6201
π Fixed flakiness in Message Test: test had not been updated after jitter functionality was added #6283
π Fix flakiness in TestConnectTimeout #6188
π Fix TLS test for Go 1.13+ #6185
π Fixed linter errors #6416 #6364 #6307
β More dollar-sign tests #6363
VTExplain: use DiscoveryGateway (and old healthcheck) until migration can be completed #6248
π Deprecated RPCs from 6.0 have been deleted #6380
β‘οΈ Update references to orchestrator repo to point to openark instead of github #6360
π Terminology: deprecate or replace references to "slave" with βreplicaβ except where referring to actual sql commands or fields returned from MySQL. #6428 #6392 #6379
π» Terminology: replace user-facing references to "slave" with βreplicaβ or βreplicationβ in UI #6481
π Fix log format - use Infof instead of Info #6413
β Cleanup temp files used in tests #6400
β Add unit test for func stringMapToString #6280
β Check errors in etcd2topo unit test #6128
β Unit test for atomic.go #6120
β Unit test for ResolveIPv4Addrs #6230
β Enabled set statement tests #6167
Replaced VTTablet ExecuteBatch api call with Execute for autocommit query. #6407 #6431
π Fixed flaky TestQueryPlanCache #6454
π Fix test compile errors #6434
β Scatter_conn unit test using new healthcheck #6458
β Test that vtexplain works with healthcheck #6252
π Tests: Removed error check that caused spurious test failures #6462
More test cases for IGNORE_MAX_PAYLOAD_SIZE #6467
β VSCopy Test: Mods/bug fixes/test logging to make test repeatable/debuggable, fix flakiness #6341 #6235
Cleanup scatter conn #6257
Simplify vtgate executor #6256
β Add tests for SHOW COLUMNS #6192
β Added fuzzer #6175
Simplified TxPool and ConnPool code #6150
-
v7.0.0-beta Changes
July 21, 2020Incompatible Changes
π VTGate: The default gateway ("-gateway_implementation" flag value) is now "tabletgateway". The old default was "discoverygatewayβ. To use the old gateway, specify the flag in the command line. Anyone already using the flag will see no change; to use the new gateway, remove the -gateway_implementation flag. The new gateway is simpler and uses a new healthcheck module with no loss of functionality. #6155
β VTGate: VTGate incorrectly returned Warning Code field name as Type. This has now been fixed. Code or scripts that depend on the incorrect behavior might break. #6437
π Deprecations
π The following vtctl commands have been deprecated. Deprecated commands will continue to work until the next release (8.0). #6428
StartSlave -> Replaced with StartReplication
StopSlave -> Replaced with StopReplication
π ChangeSlaveType -> Replaced with ChangeTabletType
Various vtctl commands had a flag called "-wait_slave_timeout". It has been replaced with β-wait_replicas_timeoutβ. Scripts using the old flag will continue to work until the next major release (8.0). #6428
Helm chart (more details to be added)
Consul (more details to be added)
Replaced VTTablet ExecuteBatch api call with Execute for autocommit query. #6407 #6431
π Bugs Fixed
VTGate / MySQL compatibility
VTGate: When using user-defined variables that have not been initialized first, a non-fatal panic occurs and the connection is dropped. #6294 #6308
VTGate: Setting system variable panics vtgate. #6162 #6163
regression: USE does not work from mysql client. #6153 #6157
π USE does not return an error if a keyspace doesn't exist. #3060 #6157
Dotted table names are not escaped properly in lookup vindex flow. #6142 #6145
Error 1105 instead of 1062 when inserting duplicate into consistent_lookup vindex. #6144
VTGate: Rewriting expressions fail when used as function parameters. #6281
β‘οΈ Rollback after error clobbers updates. #6285
Queries should not be wrapped into transactions if autocommit=1 is set. #6264
Prepared statements with bind variables in column specification produce βmissing bind varβ error. #6287 #6298
π Prepared statement queries returning NULL results are not parsed. #6240 #6249
0οΈβ£ DEFAULT function not working with status column. #6221 #6224
VTGate panic kills connection when selecting database() when using OLAP workload. #6200 #6205
Wrong column value can be inserted due to parameterization variable confusion. #6266 #6270
π Fix owned table name formatting and duplicate error code in consistent lookup. #6145
π Support for case-insensitive user defined variable. #6225
π Fix performance regression on VTGate. #6203
Return NULL for non-existent user-defined variables. #6308
π Lock function support. #6370
VTGate: MySQL connection closing should not inflate query counts. #6226
π VTGate: Show vitess_shards intermittently returns empty list. This has been fixed #5038 #5189
Other
β± VTTablet: Etcd query timeout because auto-committable transactions are not properly rolled back. #6258 #6299
π VTTablet: Fixed race in schema engine during notification of schema changes. #6268
π VTTablet: Reloading ACL config was throwing misleading error messages.his has now been fixed and the error messages improved. #6309
π VTTablet: Panic during backup because of an error in the pgzip library has been fixed by contributing a fix to pgzip and upgrading to a new release that includes the fix. #6189
VTTablet: Improvement of error handling by retrying while uploading to S3. #6182 #6395 #6405
π VTTablet: builtinbackupengine was uploading a manifest despite errors from S3. Error collection during backup has been fixed to address this. #6349 #6350
VTTablet: Fix dba connection leak in LockTables. #6424
VTTablet: Fix connection pool leak in schema engine. #6426
VTTablet: Fix panic from binlog_streamer when β-watch_replication_streamβ is set., #6253
π» Vtctld UI: Change all external links in vitess to use https #6170 #6169
Ensure SrvVSchema gets rebuilt for all cells #6276
π Fix MariadbGTIDSet multi-domain support #6184
mysqlctl: Fix connection leak during killConnection() #6245
π Fix release script to install examples in the correct directory #6159
Functionality Added or Changed
VTGate/ MySQL compatibility
π Transactions on read-only replicas are now supported. Start a transaction by first choosing the desired tablet_type ("use @replica"), then issuing a βBEGINβ statement. Internally, this will issue a βSTART TRANSACTION READ ONLYβ on the underlying database. This is only available if using tabletgateway. #6166 #6244
π Create better plans for NULL comparisons. Instead of using a SelectScatter, now vtgate will produce empty results when the predicate is a comparison against literal NULLs. #6133 #6425
β Add EXPLAIN support for vtgate plans. View the produced vtgate query plan by using EXPLAIN FORMAT=VITESS.
β Added savepoint support. #6355 #6404 #6412 #6414
"ignore_nulls" option to lookup vindexes #6147 #6222
β Add support for row_count() function. #6174 #6179
π XOR Operator is now supported in sql WHERE clause. #6369 #6371
Allow 'binary' for character_set_results. It is equivalent to null. #6237
π Improve logging by adding keyspace, table, and tablet_type. #6282
π° VTGate now takes in optional flags for βmax_payload_sizeβ and βwarn_payload_sizeβ. A payload size above the warning threshold will only produce a warning. A payload size that exceeds βmax_payload_sizeβ will result in an error. The default behavior is not to check payload size, which is backwards-compatible. #6143 #6375
A new comment directive (β/*vt+ IGNORE_MAX_PAYLOAD_SIZE=1 */) has been added to provide the ability to override the payload size checking per query. #6143
Support βselect * from information_schema.processlistβ. #6310
π VTGate: Support batch lookup queries for integer and binary types. Other types will continue to require one query per row. #6420
π Support system settings at session level. #6149
Prepare for future features by adding reserved connections. This is the ability for a connected client to have a dedicated connection instead of using the connection pool. #6303 #6313
π Allow charset introducers with vindex column values. #6236 #6314
π Support Union All between different shards. #6351
VTGate: Exclude KeyspaceId(s) from plan cache key.This improves the cache hit rate. #6233
VTGate: Reload TLS config on SIGHUP. #6215
VTGate: Drop vindex DDL. #6151
VTGate: Show vitess_tablets now reports master_term_start_time. We ensure that this is reported correctly after reparents. #6135 #6292 #6293
VReplication
VStream Copy extends the VStream API to stream entire databases or tables. Earlier you specified a position to start streaming events from. #6277
0οΈβ£ VReplication workflows now accept "-cells" as an option. This can be a comma-separated list of cells or cellsAliases. The source tablet for vreplication will be chosen randomly among the tablets that belong to the specified cells/cellsAliases. If the option is not provided, we default to the cell that is local to the vreplication target. #6442, #6456
Schema Versioning. Tablets now have a schema tracker that can be enabled to store schema versions as DDLs are applied. The _vt.schema_version table contains the schema with the corresponding ddl and gtid position when the ddl was applied. The tracked schemas are used by the new Historian service to provide vstreams with the correct schema while streaming old events. #6300
π Foreign Keys are now supported on the target database during vreplication. In the copy phase we disable FK checks and re-enable them once the replication phase starts. #6284
Materialize: added a drop_constraint option during table copy to strip constraints while creating target tables #6422
π Materialize: optimizations while creating target tables resulting in a significant improvement in performance when moving a large number of tables from an external MySQL instance #6207, #6251
β¬οΈ DropSources: Added β-rename_tablesβ option to rename the source tables rather than dropping them entirely. #6383
β Add keyspace, shard and workflow labels to VReplicationSecondsBehindMaster metrics #6279
π» Correct command line help #6187
Point-in-time Recovery
π In the 4.0 release we introduced functionality to recover a keyspace into a snapshot based on a specified time. That feature only allowed you to restore a backup, but did not have the ability to replay binlogs up to the specified time. The feature has now been completed so that true point-in-time recovery is possible. This is currently only supported for MySQL compatible installations, not MariaDB. #4857 #6267 #6408
Healthcheck
π VTGate can now use a new healthcheck that is logically simpler while providing the same functionality. This is the default. The older healthcheck has been deprecated but is still available to use if desired by setting "-gateway_implementation=discoverygateway" #5750 #6155
VTCombo has been migrated to use the new healthcheck #6302
MiniVitess
π³ A docker container that can be used to bring up a self-contained vitess cluster, which can automatically resolve and self-configure to meet an external MySQL cluster. While this might sound very similar to VTCombo and vttestserver, it is not quite the same. MiniVitess runs in docker, brings up distinct processes for each vitess component and works with an external MySQL cluster. #6377
Examples / Tutorials
Vitess operator example #6154
Region sharding example #6209 #6275
π Change local example to use new command DropSources #6178
β Are-you-alive example now makes it easier to test multiple endpoints #6202
π³ Local docker tutorial added to website
Other
β‘οΈ Vtctl ListAllTablets now shows the MasterTermStartTime. This is shown as "" for read-only tablet types. In case of an old master that has not yet updated the topo record, TabletType will be reported as βunknownβ #6135
π New debug handlers have been added to set block and mutex profiling. #6137
0οΈβ£ Vitessdriver: Allow overriding driver name. Default stays as "vitess" #6138
β VTCombo: add options to allow starting MySQL within VTCombo. Note that vttestserver provides similar functionality. #6265
β Disable foreign key checks during preflight schema change tests #4696
π Fix labels for opentsdb #6289
π Allow empty shards when running ValidateSchemaKeyspace behind flag β-skip-no-masterβ #6216
Death by SIGPIPE on stdout/stderr can be avoided by setting the new flag βcatch-sigpipeβ #6297
π Field query support in shard targeting #6132
π Documentation
π Fix links to local example and sample client in vitessdriver godoc #6254
π Flag documentation has been improved #6261
π All flags are now documented in the program reference on the website.
π Build Environment Changes
π golangci-lint now runs as a pre-commit hook and CI check #6385 #6406
β CI test for region_sharding example #6275
β Removed staticcheck as a pre-commit hook because it is included in golangci-lint #6416
β Faster unit tests #6415
β¬οΈ Consul-api version upgraded to 1.8.0 #6358
β¬οΈ Upgrade pgzip to v1.2.4 #6189
β¬οΈ Upgrade log4j2 version to 2.13.3 #6403
β Add docker_local target to Makefile and a docker/local/run.sh script #6393
π§ Build: Only check for k3s on Linux #6353
Functionality Neutral Changes
β Add context to MysqlDaemon schema functions. This lets us handle disconnected RPCs instead of the client hanging forever. #6243
π¨ Refactor table filtering logic #6242 #6259
π¨ VTGate: Replace Sprintf with string concatenation in normalizer, this reduces CPU usage on benchmarks #6127
VTTablet: Reduce the number of packets sent from vttablet to MySQL #6130
π Fixed flakiness in External Connector Test: fixed race #6201
π Fixed flakiness in Message Test: test had not been updated after jitter functionality was added #6283
π Fix flakiness in TestConnectTimeout #6188
π Fix TLS test for Go 1.13+ #6185
π Fixed linter errors #6416 #6364 #6307
β More dollar-sign tests #6363
VTExplain: use DiscoveryGateway (and old healthcheck) until migration can be completed #6248
π Deprecated RPCs from 6.0 have been deleted #6380
β‘οΈ Update references to orchestrator repo to point to openark instead of github #6360
π Terminology: deprecate or replace references to "slave" with βreplicaβ except where referring to actual sql commands or fields returned from MySQL. #6428 #6392 #6379
π Fix log format - use Infof instead of Info #6413
β Cleanup temp files used in tests #6400
β Add unit test for func stringMapToString #6280
β Check errors in etcd2topo unit test #6128
β Unit test for atomic.go #6120
β Unit test for ResolveIPv4Addrs #6230
β Enabled set statement tests #6167
-
v6.0.20-20200818 Changes
August 18, 2020 -
v6.0.20-20200624 Changes
June 24, 2020π This update fixes a regression which was significant enough that it was decided to backport to the release branch.
π Bug fixed
π§ #6250 added support for float64 durations in vttablet's yaml configuration. The change introduced a regression where the heartbeat interval unit was changed to seconds, but the conversion was not correspondingly changed. In other words, a heartbeat interval of 1s would have resulted in a value of 1000s instead. The net effect is that healthy replicas would be reported as heavily lagged.
This issue affects only those who enable heartbeats.
π This regression as merged into https://github.com/vitessio/vitess/releases/tag/v6.0.20-20200617.
π #6359 fixes #6356