Problem/Motivation

drupal_phpunit_find_extension_directories considers every file in the file system, we can use our existing filter to speed up matching .info.yml files.

Proposed resolution

Attempt to utilise RecursiveExtensionFilterIterator to cut down on discovery time in phpunit.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

drupal_phpunit_find_extension_directoriesa

CommentFileSizeAuthor
#2 3050881-speed-1.patch2.06 KBlarowlan

Comments

larowlan created an issue. See original summary.

larowlan’s picture

Status: Active » Needs review
StatusFileSize
new2.06 KB

Before: Time to boostrap class-loader: 11826.57 ms
After: Time to boostrap class-loader: 526.51 ms

Before/after: 694 directories

dawehner’s picture

index 82a1830547..d956ff0e1f 100644
--- a/core/tests/bootstrap.php

--- a/core/tests/bootstrap.php
+++ b/core/tests/bootstrap.php

@@ -8,7 +8,9 @@
 use Drupal\Core\Composer\Composer;
+use Drupal\Core\Extension\Discovery\RecursiveExtensionFilterIterator;
 use PHPUnit\Runner\Version;

@@ -22,7 +24,9 @@
 function drupal_phpunit_find_extension_directories($scan_directory) {
   $extensions = [];
-  $dirs = new \RecursiveIteratorIterator(new \RecursiveDirectoryIterator($scan_directory, \RecursiveDirectoryIterator::FOLLOW_SYMLINKS));
+  $filter = new RecursiveExtensionFilterIterator(new \RecursiveDirectoryIterator($scan_directory, \RecursiveDirectoryIterator::FOLLOW_SYMLINKS | \FilesystemIterator::CURRENT_AS_SELF), []);
+  $filter->acceptTests(TRUE);
+  $dirs = new \RecursiveIteratorIterator($filter);

This certainly brings phpunit a bit nearer to what \Drupal\simpletest\TestDiscovery::registerTestNamespaces is doing: Using our own extension discovery. I'm wondering conceptually, is it okay to have tests relying on code which will be tested by it? I think it is worth, given the huge speed difference! I think for running on the testbot it won't make a lot of difference right now, as this all still goes through run-tests.sh, but for the local development this method brings huge benefits.

@larowlan
It feels like this is not really a bug, rather a task, any reasoning for this?

  1. +++ b/core/tests/bootstrap.php
    @@ -22,7 +24,9 @@
    -  $dirs = new \RecursiveIteratorIterator(new \RecursiveDirectoryIterator($scan_directory, \RecursiveDirectoryIterator::FOLLOW_SYMLINKS));
    +  $filter = new RecursiveExtensionFilterIterator(new \RecursiveDirectoryIterator($scan_directory, \RecursiveDirectoryIterator::FOLLOW_SYMLINKS | \FilesystemIterator::CURRENT_AS_SELF), []);
    

    Should there be a @see \Drupal\Core\Extension\ExtensionDiscovery::scanDirectory maybe? Is there a reason we don't pass along all the flags?

        $flags = \FilesystemIterator::UNIX_PATHS;
        $flags |= \FilesystemIterator::SKIP_DOTS;
        $flags |= \FilesystemIterator::FOLLOW_SYMLINKS;
        $flags |= \FilesystemIterator::CURRENT_AS_SELF;
  2. +++ b/core/tests/bootstrap.php
    @@ -49,22 +53,16 @@ function drupal_phpunit_contrib_extension_directory_roots($root = NULL) {
    -    $paths[] = is_dir("$path/modules") ? realpath("$path/modules") : NULL;
    ...
    +    $paths[] = "$sites_path/$site";
    

    I wonder, do we need to call realpath on these files, like previously?

lendude’s picture

Issue tags: +Needs manual testing

Was just scanning if we have any test coverage for this and looks like we don't, see comment in \Drupal\Tests\Core\Test\StubTestSuiteBase::findExtensionDirectories

So manual testing for this I guess

lendude’s picture

Component: simpletest.module » phpunit
larowlan’s picture

Category: Bug report » Task

It feels like this is not really a bug, rather a task, any reasoning for this?

yep, you're right

Will address the reviews

mxr576’s picture

mxr576’s picture

FYI, I am working on a patch in #2943172 that uses ExtensionDiscovery to discover test files.
This comment describes the motivation why ExtensionDiscovery should be used instead of RecursiveExtensionFilterIterator: https://www.drupal.org/project/drupal/issues/2943172#comment-13210705

Maybe this issue could be closed as a duplicate if you agree with the approach in the other issue.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

larowlan’s picture

Status: Needs review » Closed (duplicate)

Closing in favour of #3223881: Speed up test discovery with \RecursiveCallbackFilterIterator which doesn't bring the 'tests relying on code they're testing' issue