DataTongs for MySQL

From the directory containing your DataTongs.settings.json:

DataTongs

With a custom config file:

DataTongs --ConfigFile:/etc/schemasmith/extract-reference-data.json

With a full MySQL connection string override:

DataTongs --ConnectionString:"Server=myserver;Port=3306;Database=AppMain;Uid=appuser;Pwd=secret;"

DataTongs connects to the source database, reads every table listed in the Tables array, serializes each row to JSON in deterministic key order, and writes a sync script plus a sibling .tabledata file for each table. Progress is logged to DataTongs - Progress.log and any errors to DataTongs - Errors.log in the working directory.

Table name

Specify tables by name. If you omit the schema, the connection's Database is assumed:

{ "Name": "Country" }
{ "Name": "Products" }
{ "Name": "config.feature_flags" }

DataTongs validates that each table exists in the source database before attempting extraction. Tables that don't exist are skipped with an error message.

Key columns

Key columns define the row-matching predicate — the unique-key match that drives MySQL's ON DUPLICATE KEY UPDATE behaviour.

Auto-detection. When KeyColumns is blank, DataTongs queries the table's indexes and selects the primary key if one exists. If there's no primary key, it falls back to the first available unique index. Nullable columns in the detected key are automatically prefixed with * for NULL-safe comparison. This handles the vast majority of tables without any manual configuration.

Manual override. When you specify KeyColumns, DataTongs uses your list instead of auto-detection. Separate multiple columns with commas:

{ "Name": "OrderLine", "KeyColumns": "OrderID, LineNumber" }

Nullable key columns. If a key column allows NULLs, prefix it with *. This generates NULL-safe matching for the change-detection predicate:

{ "Name": "Mapping", "KeyColumns": "SourceCode, *TargetCode" }

When auto-detection discovers a nullable column in a unique index, it automatically applies the * behaviour. Note: on MySQL, INSERT ... ON DUPLICATE KEY UPDATE itself matches on the table's declared unique keys, so a column prefixed with * must still participate in a unique index for the merge to work as intended.

No key available. If a table has no primary key, no unique index, and no KeyColumns configured, DataTongs skips the table with an error message.

Filter

The Filter field accepts a SQL WHERE clause (without the WHERE keyword). It controls which rows are extracted from the source database.

{ "Name": "FeatureFlags", "KeyColumns": "FlagName", "Filter": "IsActive = 1" }

This extracts only active flags. On MySQL, full delete-sync semantics are not available via INSERT ... ON DUPLICATE KEY UPDATE, so Filter acts purely as an extraction constraint on this platform (see ShouldCast flags — MergeDelete).

MySQL doesn't have a MERGE statement. DataTongs generates the appropriate idiom based on MergeUpdate and MergeDelete: INSERT ... ON DUPLICATE KEY UPDATE for insert + update, INSERT IGNORE for seed-only, and INSERT ... ON DUPLICATE KEY UPDATE followed by DELETE WHERE NOT EXISTS for full sync. The script is self-contained — no staging tables, no external files, no bulk-load dependencies. Run it against any compatible database and the data arrives.

INSERT ... ON DUPLICATE KEY UPDATE

INSERT INTO `northwind`.`shippers` (`company_name`, `phone`, `shipper_id`) VALUES
  ('Speedy Express',   '(503) 555-9831', 1),
  ('United Package',   '(503) 555-3199', 2),
  ('Federal Shipping', '(503) 555-9931', 3)
ON DUPLICATE KEY UPDATE
  `company_name` = VALUES(`company_name`),
  `phone`        = VALUES(`phone`);

The ON DUPLICATE KEY UPDATE clause runs whenever an incoming row would collide with an existing unique-key value, carrying the new column values from the VALUES list via the VALUES() function. Primary-key and unique columns appear in the INSERT list but not in the UPDATE clause, so their values are set on insert and never modified on collision.

INSERT IGNORE (seed only)

INSERT IGNORE INTO `northwind`.`shippers` (`company_name`, `phone`, `shipper_id`) VALUES
  ('Speedy Express',   '(503) 555-9831', 1),
  ('United Package',   '(503) 555-3199', 2),
  ('Federal Shipping', '(503) 555-9931', 3);

When MergeUpdate is false, DataTongs emits INSERT IGNORE. Rows that would collide with an existing unique key are silently skipped; only missing rows are inserted.

Full-sync delete step

When both MergeUpdate and MergeDelete are on, DataTongs appends a delete step after the insert/update to remove any target rows not present in the source data:

DELETE FROM `northwind`.`shippers`
 WHERE `shipper_id` NOT IN (1, 2, 3);

When a Filter is configured, the delete step is scoped to rows matching the filter, so rows outside the filter are never removed.

Auto-increment columns

Auto-increment columns are handled at the INSERT level — explicit values from the source are carried into the VALUES list so existing auto-increment sequences are preserved. Generated (virtual or stored) columns are auto-excluded from extraction because they cannot be assigned directly.

DataTongs can do more than write scripts — it can wire the extracted data straight into your schema package's declarative delivery pipeline. Pass --ConfigureDataDelivery and DataTongs writes a DataDelivery block into each matching table's JSON file after extracting the data.

DataTongs --ConfigureDataDelivery
DataTongs --ConfigureDataDelivery --TemplatePath:./Templates/Main

The switch can also be enabled via configuration:

{
  "ShouldCast": { "ConfigureDataDelivery": true }
}

What it does

For every table that produces output, DataTongs opens Tables/<table>.json inside the template and writes (or overwrites) a DataDelivery block with:

Tables whose JSON file doesn't exist in the template are skipped with a warning. No file is created from scratch — this tool configures existing tables, it doesn't scaffold new ones (use SchemaTongs for that).

Default MergeType derivation

When a table's config doesn't set MergeType explicitly, the default is derived from the global ShouldCast flags:

A per-table MergeType on the Tables[] entry always wins over the derived default.

Template root discovery

--ConfigureDataDelivery needs to know where your template lives. It uses this precedence:

1. --TemplatePath:<path> CLI switch.
2. TemplatePath config key.
3. Auto-discovery: walk upward from ContentPath looking for the nearest directory containing a Template.json.

If nothing is found, DataTongs logs a warning and skips the configuration step — extraction still completes normally. See Schema Packages — DataDelivery for the full property reference and the schema validation that catches misconfigurations at authoring time.