Base fields versus config fields, and how to handle the latter in tests
All fields are equal in Drupal 8! Back in Drupal 7 we had two different systems for data on entities, which worked fairly differently: entity properties that were defined as database fields and controlled by hardcoded form elements, and user-created fields on entities that had widgets and formatters. But now in Drupal 8, every value on an entity is a field, with the same configuration in the UI, access to the widgets and formatters, and whose data is accessed in the same way.
Is this the end of the story? No, not quite. For site builders, everything is unified, and that's great. For code that consumes data from entities, everything is unified, and that's great too. But for the developer actually working with those fields, whether a field is a base field or a config field still makes a big difference.
Config fields are easier...
Config fields are manipulated via a UI, and that makes them simple to work with.
Setup
Config fields win easily here: a few clicks in the UI to create the field, a few options to choose in dropdowns for the widget and formatter options and you're set. Export to config, commit: done!
Base fields are more fiddly here. You need to be familiar with the Entity system, and understand (or copy-pasta!) the base field definitions. Then, knowing the right widget and formatter to use and how to set their options requires close reading of the relevant plugin classes.
Update
Config fields win again: change in the UI, export to config, deploy!
For base fields, you once again need to know what you're doing with the code, and you now need a hook_update_N() to make the change to the database (now that entity updates are longer done automatically).
Removal
This is maybe a minor point, but while removing a config field is again just a UI click and a config export, removing a base field will again require a hook_update_N() to tell the Entity API to update the database tables.
...but base fields are more robust
Given the above, why bother with base fields? Well, if your fields just sit there holding data, you can stop reading. But if your fields are involved in your custom code, as a developer, this should give you an unpleasant feeling: your code in modules/custom/mymodule is dependent on configuration that lives in the site's config export.
Robustness
The fact that config fields are so easy to change in the UI is now a mark against them. Another developer could change a field without a full understanding of the code that works with it. The code might expect a value to always be there, but the field is now non-required. The code might expect certain options, but the field now has an extra one.
Instead of this brittle dependency, it feels much safer to have base fields defined closer to the code that makes use of them: in the same module.
Tests
The solution to the dependency problem is obviously tests, which would pick up any change that breaks the code's expectations of how the fields behave. But now we hit a big problem with config fields: your test site doesn't have those fields!
My first attempt at solving this problem was to use the Configuration development module. This allows you to export config from the site to a module's /config/install folder. A test that installs that module then gets those config items imported.
It's a quick and simple approach: when a test crashes because of a missing field, find it in the config folder, add it to the right module's .info.yml file, do drush cde MODULE and commit the changes and the newly created files.
This approach also works for all other sorts of config your test might need: taxonomy vocabularies, node types, roles, and more!
But now you have an additional maintenance burden because your site config is now in three places: the site itself, the config export, and now also in module config. And if developers who change config forget to also export to module config, your test is now no longer testing how the site actually works, and so will either fail, or worse, won't be actually covering what you expect any more.
Best of both worlds: import from config
To summarize: we want the convenience of config fields, but we want them to be close to our code and testable. If they're testable, we can maybe stand to forego the closeness.
My best solution to this so far is simple: allow tests to import configuration direct from the site config sync folder.
Here's the helper method I've been using:
/**
* Imports config from a real local site's config folder.
*
* TODO: this currently only supports config entities.
*
* @param string $site_key
* The site key, that is, the subfolder in /config in which to find the
* config files.
* @param string $config_name
* The config name. This is the same as the YML filename without the
* extension. Note that this is not checked for dependencies.
*
* @throws \Exception
* Throws an exception if the config can't be imported.
*/
protected function importSiteConfig(string $site_key, string $config_name) {
$storage = $this->container->get('config.storage.sync');
// Code cribbed from Config Devel module's ConfigImporterExporter.
$config_filename = "../config/{$site_key}/{$config_name}.yml";
if (!file_exists($config_filename)) {
throw new \Exception("Unable to find config file $config_filename to import from.");
}
$contents = @file_get_contents($config_filename);
$data = $storage->decode($contents);
if (!$data) {
throw new \Exception("Failed to import config $config_name from site $site_key.");
}
// This assumes we only ever import entities from site config for tests,
// which so far is the case.
$entity_type_id = $this->container->get('config.manager')->getEntityTypeIdByName($config_name);
if (empty($entity_type_id)) {
throw new \Exception("Non-entity config import not yet supported!");
}
$entity = $this->container->get('entity_type.manager')->getStorage($entity_type_id)->create($data);
$entity->save();
}
Use it in your test's setUp() or test methods like this:
// Import configuration from the default site config files.
$this->importSiteConfig('default', 'field.storage.node.field_my_field');
$this->importSiteConfig('default', 'field.field.node.article.field_my_field');
As you can see from the documentation, this so far only handles config that is entities. I've not yet had a use for importing config that's not an entity (and just about all config items are entities except for the ones that are a collection of a module's settings). And it doesn't check for dependencies: you'll need to import them in the right order (field storage before field) and ensure the modules that provide the various things are enabled in the test.
I should mention for the sake of completeness that there's another sort of field, sort of: bundle fields. These are in code like base fields, but limited to a particular bundle. They also have a variety of problems as the system that support them is somewhat incomplete.
Finally, it occurs to me that another way to bridge the gap would be to allow editing base fields in the UI, and then export that back to code consisting of the BaseFieldDefinition calls. But hang on... haven't I just reinvented Drupal 7-era Features?
UPDATE
It turned out this was crashing when importing fields with value options. This code (and that in Config Devel module, which is where I cribbed it from) wasn't properly using the Config API. Here's updated code, which incidentally, also now can handle non-entity config:
/**
* Imports config from a real local site's config folder.
*
* Needs config module to be enabled.
*
* @param string $site_key
* The site key, that is, the subfolder in /config in which to find the
* config files.
* @param string $config_name
* The config name. This is the same as the YML filename without the
* extension.
*
* @throws \Exception
* Throws an exception if the config can't be imported.
*/
protected function importSiteConfig(string $site_key, string $config_name) {
$storage = $this->container->get('config.storage.sync');
// Code cribbed from Config Devel module's ConfigImporterExporter.
$config_filename = "../config/{$site_key}/{$config_name}.yml";
if (!file_exists($config_filename)) {
throw new \Exception("Unable to find config file $config_filename to import from.");
}
$contents = @file_get_contents($config_filename);
$data = $storage->decode($contents);
if (!$data) {
throw new \Exception("Failed to import config $config_name from site $site_key.");
}
unset($data['uuid']);
// We have to partially mock the source storage, because otherwise
// SystemConfigSubscriber::onConfigImporterValidateSiteUUID() will complain
// because the source config doesn't contain a system.site config. (For
// general information, the single-import UI in core doesn't hit this
// problem because it TOTALLY cheats and pretends its source is the full
// site config! Though who knows how it manages not to trip up when the site
// config folder is empty!)
$source_storage = $this->getMockBuilder(StorageReplaceDataWrapper::class)
->setConstructorArgs([$this->container->get('config.storage')])
->setMethods(['exists'])
->getMock();
// Satisfy SystemConfigSubscriber that we have a system.site config.
$source_storage->expects($this->any())
->method('exists')
->willReturn(TRUE);
$source_storage->replaceData($config_name, $data);
// Similarly mock the storage comparer.
$storage_comparer = $this->getMockBuilder(StorageComparer::class)
->setConstructorArgs([$source_storage, $this->container->get('config.storage')])
->setMethods(['validateSiteUuid'])
->getMock();
// Satisfy SystemConfigSubscriber that system.site config is valid.
$storage_comparer->expects($this->any())
->method('validateSiteUuid')
->willReturn(TRUE);
$storage_comparer->createChangelist();
$config_importer = new ConfigImporter(
$storage_comparer->createChangelist(),
$this->container->get('event_dispatcher'),
$this->container->get('config.manager'),
$this->container->get('lock'),
$this->container->get('config.typed'),
$this->container->get('module_handler'),
$this->container->get('module_installer'),
$this->container->get('theme_handler'),
$this->container->get('string_translation'),
$this->container->get('extension.list.module')
);
$config_importer->import();
}
}
UPDATE 2
I've found an occasional problem with site config items that have third-party settings from modules that aren't relevant to the test. Examples include Menu UI settings in node types, and contrib translation settings in fields. Rather than enable extra modules in the test, these settings can be hacked out of the data after we decode it from the YAML. Here's an updated version of the code.
use Drupal\Core\Config\ConfigImporter;
use Drupal\Core\Config\StorageComparer;
use Drupal\config\StorageReplaceDataWrapper;
/**
* Imports config from a real local site's config folder.
*
* @param string $site_key
* The site key, that is, the subfolder in /config in which to find the
* config files.
* @param string $config_name
* The config name. This is the same as the YML filename without the
* extension.
* @param array $third_party_filter
* (optional) Keys in the config item's third party settings that are to be
* removed prior to import. This prevents unwanted dependencies on modules
* that are not relevant to a test. Defaults to an empty array; no keys
* removed.
*
* @throws \Exception
* Throws an exception if the config can't be imported.
*/
protected function importSiteConfig(string $site_key, string $config_name, array $third_party_filter = []) {
$storage = $this->container->get('config.storage.sync');
// Code cribbed from Config Devel module's ConfigImporterExporter.
$config_filename = "../config/{$site_key}/{$config_name}.yml";
if (!file_exists($config_filename)) {
throw new \Exception("Unable to find config file $config_filename to import from.");
}
$contents = @file_get_contents($config_filename);
$data = $storage->decode($contents);
if (!$data) {
throw new \Exception("Failed to import config $config_name from site $site_key.");
}
unset($data['uuid']);
// Remove third-party settings.
foreach ($third_party_filter as $settings_key) {
unset($data['third_party_settings'][$settings_key]);
}
// We have to partially mock the source storage, because otherwise
// SystemConfigSubscriber::onConfigImporterValidateSiteUUID() will complain
// because the source config doesn't contain a system.site config. (For
// general information, the single-import UI in core doesn't hit this
// problem because it TOTALLY cheats and pretends its source is the full
// site config! Though who knows how it manages not to trip up when the site
// config folder is empty!)
$source_storage = $this->getMockBuilder(StorageReplaceDataWrapper::class)
->setConstructorArgs([$this->container->get('config.storage')])
->setMethods(['exists'])
->getMock();
// Satisfy SystemConfigSubscriber that we have a system.site config.
$source_storage->expects($this->any())
->method('exists')
->willReturn(TRUE);
$source_storage->replaceData($config_name, $data);
// Similarly mock the storage comparer.
$storage_comparer = $this->getMockBuilder(StorageComparer::class)
->setConstructorArgs([$source_storage, $this->container->get('config.storage')])
->setMethods(['validateSiteUuid'])
->getMock();
// Satisfy SystemConfigSubscriber that system.site config is valid.
$storage_comparer->expects($this->any())
->method('validateSiteUuid')
->willReturn(TRUE);
$storage_comparer->createChangelist();
$config_importer = new ConfigImporter(
$storage_comparer->createChangelist(),
$this->container->get('event_dispatcher'),
$this->container->get('config.manager'),
$this->container->get('lock'),
$this->container->get('config.typed'),
$this->container->get('module_handler'),
$this->container->get('module_installer'),
$this->container->get('theme_handler'),
$this->container->get('string_translation'),
$this->container->get('extension.list.module')
);
$config_importer->import();
}