Changelog

All notable changes to this project will be documented in this file.

Updates should follow the Keep a CHANGELOG principles.

This project adheres to Semantic Versioning. For major version 0, we will increment the minor version for backward-incompatible changes.

Unreleased

Changed

• Moved the Csv extractor and loader to soong/csv.
• Moved the DBAL integrations to soong/dbal.
• Moved the console command implementations to soong/console.
• CountableExtractorBase::count() now supports caching of counts - extractors extending CountableExtractorBase should override getUncachedCount() if they have a more efficient means of retrieving the source data count than iterating over all the source data.
• A new PropertyList interface is added, which the Extractor and Loader interfaces extend instead of defining their own getProperties() and getKeyProperties() methods.
• The processing of each record during migration is now implemented as a pipeline through all transformers and the loader, with the new RecordPayload class (containing the source record and destination record) as the payload passed through each. As a result, RecordTransformer::transform($sourceData, $resultData) becomes RecordTransformer::__invoke($recordPayload), and Loader::load($resultData) becomes Loader::__invoke($recordPayload).
• Property transformers are now processed as a pipeline, and implement __invoke($data) rather than transform($data).
• The Operation interface, with concrete implementations MigrateOperation and RollbackOperation, has been added. These classes, containing the Task instance being operated on, are now the invokable stages of the task pipeline - operations are no longer methods in the Task objects.
• TaskPipeline has been renamed to TaskCollection to better reflect what it is. For better compatibility with standard collection terminology, the getTask/getAllTasks/addTask methods have been renamed to get/getAll/add.
• Classes with the same name as the interface they are implementing have been renamed to include the word “Basic”. Specifically:
  • Soong\Data\Record -> Soong\Data\BasicRecord
  • Soong\Data\RecordFactory -> Soong\Data\BasicRecordFactory
  • Soong\Task\EtlTask -> Soong\Task\SimpleEtlTask
  • Soong\Task\Task -> Soong\Task\SimpleTask
  • Soong\Task\TaskCollection -> Soong\Task\SimpleTaskCollection
• The League\Pipeline package is now used to run migration tasks. No API changes - new dependency in composer.json.

0.7.0 - 2019-06-25

Changed

• The former Transformer interface is now named PropertyTransformer, to distinguish it from the new RecordTransformer interface.
• Property transformer classes have been moved to the Soong\Transformer\Property namespace.
• The structure of sample task configurations has been changed to use an array of RecordTransformer instances under the transform key.
• Transformer construction has been moved from the EtlTask implementation to the EtlCommand implementation - the transform option for EtlTask now requires an array of RecordTransformer instances.
• PropertyTransformer::transform() no longer specifies the types of its argument and return value. Implementations should specify their specific return type, and validate the expected argument type.
• Record::getProperty() renamed to getPropertyValue(), setProperty() renamed to setPropertyValue().
• Symfony Console dependency has been loosened to allow versions 3.4 through 4.x.
• Release checklist added to CONTRIBUTING

Removed

• The Property and PropertyFactory interfaces and implementations have been removed - all code using properties now uses values directly, leaving type-checking to PHP. property_factory configuration options have accordingly been removed.

Added

• The RecordTransformer interface has been added.
• The PropertyMapper record transformer class has been added.
• The Copy record transformer class has been added.
• The --limit option has been added to the migrate and rollback commands.
• Transformer exceptions added.

0.6.0 - 2019-05-01

Changed

• EtlTask now accepts its extract, key_map, and load components as object instances rather than constructing them from configuration.
• DataProperty interface renamed to Property, and DataRecord interface renamed to Record.
• Derivatives of ExtractorBase now must accept a record_factory configuration option, which is an instance of RecordFactory.
• EtlTask replaced the string record_class with RecordFactory instance record_factory.

Added

• The Filter interface has been added, to determine whether a DataRecord should be processed.
• The Select filter has been added, allowing for filtering by comparing DataRecord properties to values using PHP comparison operators.
• The --select option has been added to the migrate command, allow for ad-hoc filtering of extracted data at runtime.
• PropertyFactory and RecordFactory interfaces/classes added for creation of Property and Record instances.
• Added basic console command tests.
• property_factory configuration option added to EtlTask, LoaderBase.
• ExtractorException, KeyMapException, and LoaderException classes added.
• Unit test for Record added.

0.5.3 - 2019-04-12

Changed

• Things are now configured to generate the API documentation using Doxygen on readthedocs - the generated docs are no longer kept in the repo.

0.5.2 - 2019-04-05

Changed

• addTask now takes an existing Task object instead of a class and configuration.
• Static create() methods removed from all components and constructors made public.
• Static methods removed from Task component and moved to non-static methods on the new TaskPipeline component: addTask(), getTask(), getAllTasks().

Added

• ConfigurableComponent interface added, and all configurable component interfaces inherit from it.
• OptionsResolverComponent added implementing ConfigurableComponent using Symfony\OptionsResolver - this is now the base class for all configurable components. Any such components adding configuration options to their parent class must implement optionDefinitions() to defined them.
• Commands now use hassankhan/config instead of custom YAML handling - configuration now can be YAML, JSON, or XML transparently (examples provided for each).
• TaskPipeline component for managing groups of tasks.
• ComponentNotFound and DuplicateTask exceptions added.
• Tests for Extractor, KeyMap, Loader, and Task components.
• Tests for KeyMapLookup component.
• Smoke test to make sure all provided examples keep working.

Removed

• isCompleted method on Task - unneeded until we add dependencies.

0.4.0 - 2019-02-15

Added

• EtlTaskInterface::getLoader()
• Tests for Data and Transformer components.

Changed

• DBAL and Csv implementations have been moved:
  • Soong\Csv\Extractor -> Soong\Extractor\Csv
  • Soong\Csv\Loader -> Soong\Loader\Csv
  • Soong\DBAL\Extractor -> Soong\Extractor\DBAL
  • Soong\DBAL\KeyMap -> Soong\KeyMap\DBAL
  • Soong\DBAL\Loader -> Soong\Extractor\Loader
• Interface and Trait suffixes removed from all interfaces and traits.
• All interfaces moved into Contracts directory.
• All main components must now be created using Class::create() rather than new. This affects:
  • DataPropertyInterface
  • DataRecordInterface
  • ExtractorInterface
  • KeyMapInterface
  • LoaderInterface
  • TaskInterface
  • TransformerInterface
• Explicit component documentation pages have been removed in favor of Doxygen-generated documentation.
• Existing inline documentation has been cleaned up and expanded.

Removed

• KeyMapFactory
• MemoryKeyMap

0.3.0 - 2019-02-05

Added

• Added getExtractor() and getAllTasks() to task interfaces/implementations.
• Initial implementation of the status console command.
• All documentation moved into this repo, will be available on readthedocs.io.
• DataPropertyInterface::isEmpty()
• DataRecordInterface::propertyExists()

Removed

• Removed subtask concept from task interfaces/implementations.
• Removed CountableExtractorInterface.

Changed

• DataRecordInterface::setProperty() no longer accepts null values (only property objects with null values).
• DataRecordInterface::getProperty() no longer returns null values (only property objects with null values).
• TransformerInterface::transform() no longer accepts or returns null values (only property objects with null values).
• The $configuration parameter to TransformerInterface::transform() has been removed - configuration should be passed in the constructor instead.
• SHA256 for key hashing in KeyMapBase.
• Added configuration key for hash algorithm.

Fixed

• Hashing serialized keys needs to make sure values are always strings.

0.2.1 - 2019-01-24

Changed

• Merged all the repos back into one for ease of development.

0.1.0 - 2019-01-17

Initial release on Packagist.