Drupal Planet Posts

It's true, no April fools. You can make your Browser tests run much quicker. How? By deleting them!

You will of course need to add a corresponding Kernel test - and that's the trick. Kernel tests run much faster than Browser tests.

But Browser tests make requests to the test site using an internal web browser, I hear you say, whereas Kernel tests make API calls directly. Kernel tests have their uses for testing APIs, but Browser tests are needed to test actual HTML output.

Aha! Kernel tests can now make HTTP requests.

This is subject to a number of caveats and limitations: there is no session, and forms can't be submitted. And functionality such as a current user, blocks on the page, and page caching will need additional setup.

And more generally, with Kernel tests, modules are enabled but not installed: you need to handle things like entity schemas, database tables, and install config yourself in the test. The benefit though is that you only set up the parts of the module that you need for your test.

So not all Browser tests are suitable for conversion. But a lot of them are. We're already working on converting tests in core, and as this feature has been backported to Drupal core 11.x, contrib modules can make use of it too.

The benefits to conversion are tests that run faster, so less time developing and less time waiting for CI pipelines to run, and a lower energy footprint and lower costs for drupal.org. And they're easier to debug too.

And if you haven't yet written any tests for your module, now is an excellent time to start!

Do you need help with writing PHPUnit tests, or getting started with test-driven development? I'm available for hire - contact me!

Module Builder now has its own documentation site.

This covers the many options it offers developers for fine-tuning their module code, from dependency injection to plugin inheritance, entity base fields, form elements, permissions, library asset files, and more.

Meanwhile, the latest release of Module Builder adds a feature I've wanted to implement for a very long time: when a new form section is added to add a new component (such as a plugin, hook class, or entity type), the form scrolls up to the new section that's just been added with AJAX. This makes it much clearer to understand what's just been changed, and helps with navigating around Module Builder's forms.

At LocalGov Drupal Dev Days in London earlier this month, the topic came up of releasing custom project code as contrib modules.

There were many people in the room who said they had custom code in their site codebase that they planned to release as contrib modules, but needed to find the time to get it ready. I heard people mention the work that they had left to do for this, and it sounded very familiar: generalise the functionality, remove client-specific code, remove client-specific strings.

This reminded me of a session I did at Drupal Camp London way back in 2014, on this very topic: releasing more code from your codebase, to lower the amount of custom code and share more with the community. Since then, I've gone on to release many more contrib modules, and the introduction of more powerful APIs and systems with Drupal 8 has added to what's possible, so I thought I'd revisit my thoughts on different ways to approach this. My presentation was on the 'why' as well as the 'how', but I'll assume you know that part already.

The first thing to say is that as with tests or accessibility, it's much easier to write contributable code from the start rather than rework it later.

Fundamentally though, whether to plan from the start or retrofit, the baic principle is that you want your code to be split into two layers: the contrib, and the custom. Think of it as a contrib cake with custom icing on top.

The tricky part is where to put the dividing line. It's not always clear how much of your functionality is generic and applicable to other use cases and other clients.

I always err on the side of putting too much in contrib, and offsetting the possibility that the contrib code is too specific with customisability.

But how do we actually slice it up?

Plugins

Plugins are one of the most powerful ways of switching behaviour in Drupal. Defining your own plugin type allows you to design exactly which parts of the code are handed over to the plugin, and in as many places as you want, by adding more methods to your plugin's interface.

If the methods in the plugin start to look unrelated, you can always add a second plugin type. And if the amount of boilerplate needed for a plugin type is offputting, Module Builder generates it all for you.

It's worth also considering the lesser-known sibling of attribute plugins, the YAML plugin. If you only want to change strings or parameters, then you can put all that into YAML instead of a whole class. (And YAML plugins do allow custom classes for oddball cases.)

With a plugin system, you need a way to set the plugin to use. There are two ways you could do this: if it's a single plugin that you select, use a plain config setting. If it's a pattern that you might want several of, use a config entity that holds the reference to the plugin. This requires a fair bit of boilerplate code, but there are examples in contrib that you can crib from, such as Flag and Action Link.

And remember that there are other systems that allow ways to select a plugin: field formatters and widgets, Views handlers for fields and filters and so on, paragraph behaviours, and more.

Twig templates

Twig templates are a great way to customise output from your module. You can change strings, rearrange elements, and add CSS classes for styling.

You'll need to define the theme hook using hook_theme(), and define the variables the template uses. Then, provide a neutral version of the template in the contrib module's /templates folder, and override it in your site's theme.

Form alteration

For forms, use hook_form_alter() to change the labels of elements and their order.

Or you can even add extra form elements, and handle their values in a custom submit handler.

If your alterations start to get too complex, consider using a plugin that you pass the form to for customization.

Overridden config

Simplest of all is to use config to override values, whether they are strings or parameters.

Define the config schema for the settings, add a default config to the module's config/install, and then override it in your project's config.

Other APIs

It's worth looking at existing APIs that allow a custom module or theme to alter functionality. For example, in core, field widgets can be altered with hook_field_widget_single_element_form_alter() and field formatters with hook_field_formatter_third_party_settings_form(). And all sorts of unspeakable things can be done to Views with field, filter, argument, and sort handlers, and display extender plugins.

Shortcuts

If you're short on time and resources to work on splitting your cake up, there are some shortcuts you can take. It's what I call the 'code and run' method of releasing code: the contrib module is incomplete, but released in the hope that the next person who finds it useful will pick it up and move it forward.

• Skimp on UI: If they're settings that you don't need to override in your project, you could even make the values constants somewhere. The main thing is to make it easy to find all occurrences of a value, so that a future contributor can replace them with a value from config.
• Skimp on features: Leave space for other cases you can envisage, but don't need right now.

My opinion on this is that releasing some code, even if it's half-baked, is better than not releasing code at all, as long as you clearly explain on the project page that the module has things missing or incomplete, and leave a trail in the code in the form of comments and placeholders.

Do you need help with preparing custom code to be released as a contrib project? It's a great way to get more presence for you or your organisation. I'm available for hire - contact me!

Rector is a really powerful tool for making refactoring changes to your codebase. It's easy to use, but it's not obvious, and a lot of the documentation and articles about it are outdated or incomplete. For instance, when you go to the project page (https://www.drupal.org/project/rector) there's no clear indication of how to install it!

More and more of the code changes needed to keep your modules up to date with Drupal core are being written as Rector rules. I wrote recently about converting plugins to PHP attributes; the other big change in Drupal at the moment is hooks changing from procedural functions to class methods.

Here's the steps I took to convert the hooks in the Computed Field module:

1. Install Rector in your project. As mentioned earlier, finding the installation instructions is not obvious: they're in the github project:

composer require --dev palantirnet/drupal-rector
cp vendor/palantirnet/drupal-rector/rector.php .

This puts a rector.php file in your project root. What to do with this isn't immediately obvious either, but fortunately, in the PR for OO hook conversion there is sample code. The key part is this:

  $rectorConfig->rule(\DrupalRector\Rector\Convert\HookConvertRector::class);

You can then run Rector on your code. Remember to commit any existing changes to git first: this Rector rule changes a lot, and it's good to be able to revert it cleanly if necessary.

vendor/bin/rector process path/to/my_module

This does the conversion: hook implementation code is copied to methods in new Hook classes, and the existing hook implementations are reduced to legacy wrappers.

However, the code is all formatted to ugly PHP PSR standards. Import statements in .module file for use inside hook code will also remain. So we turn to PHPCS, which can re-format the code correctly and clean up the imports. I chose to target just the .module file and the Hook classes:

vendor/bin/phpcbf --standard=Drupal --extensions=php,module path/to/my_module/src/Hook
vendor/bin/phpcbf --standard=Drupal --extensions=php,module path/to/my_module/my_module.module

At this point, you should run your tests to confirm everything works, but the conversion should be complete.

You can of course now choose to do further refactoring on your hooks class, such as splitting it into multiple classes for clarity, moving helper functions into the class, or combining multiple hooks.

Attributes are here and they're great. I hated annotations; they were a necessary evil, but putting working code into comments just felt wrong.

But the conversion work is still ongoing: many contrib modules (and perhaps custom ones too) need to convert their plugin types to being attribute-based. There is time to do this: code in \Drupal\Core\Plugin\Discovery\AttributeDiscoveryWithAnnotations announces that attributes on plugins are required from Drupal 12, with the old-style annotation allowed to remain alongside it for backwards-compatibility until Drupal 13.

But the sooner you convert plugin types to attributes, the sooner you benefit from imported classes in definition values, the ability to put comments within the annotation, cleaner translation of labels, and use of PHP constants in attribute keys or values. And IDE autocompletion of the plugin definition keys, if you're not using Module Builder to generate your plugins (why, though? why?).

With Drupal Rector you can convert individual plugins from annotation to attribute, but to convert a plugin type, you need more than that.

Module Builder can help with this with its Adoption feature. This adds items to your module config based on the existing code, which allows you to alter or add to the code definition before re-generating it. (This feature is particularly useful for adding further injected services to an existing class such as a plugin or a form.)

Release 4.5.4 of Drupal Code Builder (the library that powers Module Builder) added adoption of plugin types, and this opens the door to converting a plugin type from annotation to attribute. The steps are are follows:

1. Go to Administration › Configuration › Development › Module Builder.
2. If your module is not already defined in Module Builder, click 'Adopt existing module', then find your module in the list and click 'Adopt module and adopt components'. If you already have the module in Module Builder, go to your module's 'Adopt' tab.
3. On the adopt components form, select the plugin types you want to convert.
4. Click 'Adopt components'.
5. Go to the 'Plugins' tab of your module. Your plugin types should be in the form.
6. Change the discovery type of each one to 'Attribute plugin'. The values will carry across (this is actually a problem with FormAPI that I've never managed to figure out, but it's actually quite handy).
7. Go to the Generate tab.
8. Select the files to write. For each plugin type, you'll want the plugin manager, and the new attribute class.

You should verify the new attribute class. Properties will have been adapted from the annotation class but they may not all be correct. In particular, default values for optional properties aren't yet handled by Module Builder, so you'll need to add those.

Once you've tidied up the attribute class, your plugin type is now attribute-enabled. (You should run the tests that cover plugins just to make sure everything is fine.)

The plugins of this type in your module are still using annotations, so now we turn to Rector.

There is full documentation on converting plugins but the gist of it is this:

$rectorConfig->ruleWithConfiguration(\DrupalRector\Drupal10\Rector\Deprecation\AnnotationToAttributeRector::class, [
  new \DrupalRector\Drupal10\Rector\ValueObject\AnnotationToAttributeConfiguration(
    '10.0.0',
    '10.0.0',
    'MyPluginType',
    'Drupal\my_module\Attribute\MyPluginType',
  ),
]);

Run rector on just your plugin folders to make it go quicker, since you know where the plugin files are:

vendor/bin/rector process path/to/module/src/Plugin/MyPluginType

That converts all the annotations, but unfortunately, it doesn't format them properly: the whole thing ends up all on one line. And PHPCS and PHPCBF don't know how to format an attribute either.

But regular expressions come to the rescue, and fortunately the syntax of attributes is fairly distinct.

In your IDE, a search of '(?=\b\w+: )' replaced with '\n ' (note two spaces, for the indent) will put each property onto its own line. It won't handle array values though. And beware: it will catch the use of colons in text strings! This is an instance where using a GUI for git makes quick work: stage the fixes to the attributes, discard everything else.

That leaves just the terminal closing bracket, which you can do by hand, or cook up a regex if you have a lot of plugin classes.

My site is back up, after a brief interlude: I moved hosts, because Gandi doubled their hosting costs. I'd left it quite late to renew, because up until now it's just been a case of ticking a box and paying money, so I didn't have much time to shop around.

I chose Krystal, because they use green energy and their procedure for transferring my domain seemed simple (there was a German hosting firm I was recommended too, but that one said something about filling in a form in a PDF and it looked horrendous).

I was warned about their use of cPanel, but I went ahead anyway, and now I can say that yes, the warning was justified.

CPanel is a mish-mash of different things, none of which join up properly. So for instance, it has a feature for creating git repositories on the server and using them to deploy your code, but that bit doesn't tell you to set up ssh keys first. You have to know. Then on the git repository management page, the URL for cloning the repository is wrong. When I opened a support ticket, I got a quick response with the correct URL, but when I suggested the information in cPanel could be fixed, I was told that the URL it shows can't be changed.

It's like someone took a wheelbarrow full of random tools and tipped it all out in front of you. Nothing works together, nothing is co-ordinated.

One of them is Softaculous, a tool which promises to install any of a bunch of web apps. I tried getting it to install Drupal before I copied over my own codebase, just in case it knew some things I didn't. It installed it with the legacy folder structure: Drupal in the project root, rather than in a /web folder. I know that technically works, but I didn't feel like totally restructuring my codebase for that.

So the first real Drupal problem was the web root. It was set up as /public_html, and the documentation for git deployment said it would deploy into that folder. I opened a support ticket to ask about changing the web root to /public_html/web, for Drupal's folder structure, and I was told that it was impossible on my cheap hosting plan; get a VPS they said. But that's overkill for this site, which gets very little traffic.

That left me with two options: restructure my codebase to follow the legacy Drupal code structure, with the web root in the project root, or add some sort of redirect. The latter is fairly easy to do with an .htaccess file in the project root, once you know how. I found this in an old forum post:

# Redirect to the actual webroot.
# Because cpanel is crap.
# @see https://www.drupal.org/hosting-support/2021-09-21/unable-to-set-document-root-as-public_htmlweb
RewriteEngine on
RewriteCond %{HTTP_HOST} ^(www.)?noreiko.com$
RewriteCond %{REQUEST_URI} !^/www/
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /www/$1
RewriteCond %{HTTP_HOST} ^(www.)?noreiko.com$
RewriteRule ^(/)?$ www/index.html [L]

That worked with a simple test of a plain HTML page, and so I deployed my site. Hurrah! Except no: form submissions weren't working: submitting a form took me to an error page complaining about the redirect being external.

Fortunately, once again I found a fix, this time in a core issue. This code goes in a settings.php file. In my case, I put it in a settings.prod.php file, since I don't need it to run on my local development copy of the site.

// Needed for cPanel crapness.
// See https://www.drupal.org/project/drupal/issues/2612160#comment-11767977
if (isset($GLOBALS['request']) && '/htdocs/index.php' === $GLOBALS['request']->server->get('SCRIPT_NAME')) {
  $GLOBALS['request']->server->set('SCRIPT_NAME', '/index.php');
}
elseif (isset($GLOBALS['request']) && '/htdocs/update.php' === $GLOBALS['request']->server->get('SCRIPT_NAME')) {
  $GLOBALS['request']->server->set('SCRIPT_NAME', '/update.php');
}

The same fix needs to be done for update.php as well. There's probably a neat way to cover both cases in one go, and also any other PHP entry points (though I think statistics.php is gone?) but I'd run out of energy by then.

Drupal runs fine in a subfolder on my laptop, where I still use MAMP and run each local site in its on own folder in the web root. (Although Tome, the Drupal static site generator doesn't handle that properly, which is why I abandoned the idea of moving my site to github pages). So it's probably something caused by the Apache redirect I'd put it; but once I got it working I didn't experiment further.

So, my site is up and running again, as you can see.

But no thanks to the proprietary software that runs the hosting.

When I start writing a class that has a dependency injection, I have a clear idea about which services it needs. I generate it -- the plugin, form, controller, or service -- and specify those services.

Then nearly always, unless it's something really very simple, I find that no matter how much I thought about it and planned it, I need to add more services. Maybe remove some too.

Fortunately, because Module Builder saves the configuration of the module code you've generated, it's easy to go back to it and edit it to add more services:

1. Edit your module in Module Builder
2. Add to the injected services for your component
3. Ensure your code file is committed to version control
4. Generate the code, and write the updated version of the code file
5. Add and commit the new DI code, while discarding the changes that remove your code. (I find it helps to use a git GUI for things like this, though git add -p works too.)

But I tend to find that I make this mistake several times as the class develops, and so I adopt the approach of using the \Drupal::service() function to get my services, and only when I'm fairly confident I'm not going to need to make any more changes to DI, I update the injected services in one go, converting all the service calls to use the service properties.

I was asked yesterday at Drupal Drinks about how to do that, and it occurred to me that there's a way of doing this so after you've updated the dependency injection with Module Builder, it's a simple find and replace to update your code.

If you write your code like this whenever you need a service:

$service_entityTypeManager = \Drupal::service('entity_type.manager');
$stuff = $service_entityTypeManager->doSomething();

Then you need to do only two find and replace operations to convert this to DI:

1. Replace '^.+Drupal::service.+\n' with ''. This removes all the lines where you get the service from the Drupal class.
2. Replace '\$service_(\w+)' with '$this->$1'. This replaces all the service variables with the class property.

Up until now I'd been calling the service variables something like $entityTypeManager so that I could easily change that to $this->entityTypeManager manually, but prefixing the variable name with a camel case 'service_' gives you something to find with a regular expression.

If you want to be really fancy, you can use a regular expression like '(?<=::service..)[\w.]+' (using dots to avoid having to escape the open bracket and the quote mark) to find all the services that you need to add to the class's dependency injection.

Something like this:

$ ag -G MyClass.php '(?<=::service..)[\w.]+' -o --nonumbers --nofilename | sort | uniq | tr "\n" ", "

will give you a list of service names that you can copy-paste into the Module Builder form. This is probably overkill for something you can do pretty quickly with the search in a text editor or IDE, but it's a nice illustration of the power of unix tools: ag has options to output just the found text, then sort and uniq eliminate duplicates, and finally tr turns it into a comma-separated list.