Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2011 Jun 1.
Published in final edited form as: J Neural Eng. 2010 May 11;7(3):036007. doi: 10.1088/1741-2560/7/3/036007

ELECTROENCEPHALOGRAPHIC (EEG) CONTROL OF THREE-DIMENSIONAL MOVEMENT

Dennis J McFarland 1, William A Sarnacki 1, Jonathan R Wolpaw 1
PMCID: PMC2907523  NIHMSID: NIHMS209078  PMID: 20460690

Abstract

Brain-computer interfaces (BCIs) can use brain signals from the scalp (EEG), the cortical surface (ECoG), or within the cortex to restore movement control to people who are paralyzed. Like muscle-based skills, BCI use requires activity-dependent adaptations in the brain that maintain stable relationships between the person’s intent and the signals that convey it. This study shows that humans can learn over a series of training sessions to use EEG for three-dimensional control. The responsible EEG features are focused topographically on the scalp and spectrally in specific frequency bands. People acquire simultaneous control of three independent signals (one for each dimension) and reach targets in a virtual three-dimensional space. Such BCI control in humans has not been reported previously. The results suggest that with further development noninvasive EEG-based BCIs might control the complex movements of robotic arms or neuroprostheses.

1. Introduction

The adult CNS displays a large repertoire of adaptive behaviors acquired through practice, usually referred to as skills. These skills are normally produced by muscles. In contrast, brain-computer interfaces (BCIs) enable people to communicate or to control devices by using brain signals rather than muscles. Thus, they can help people with devastating neuromuscular disorders such as amyotrophic lateral sclerosis (ALS), brainstem stroke, cerebral palsy, and spinal cord injury (Wolpaw and Birbaumer, 2006).

Studies to date show that humans and animals can learn to use electroencephalographic activity (EEG) recorded from the scalp, electrocorticographic activity (ECoG) recorded from the cortical surface, or signals recorded within the cortex (neuronal action potentials or local field potentials (LFPs)) to control the movements of a cursor or other device in one or two dimensions (Wolpaw et al., 1991, 2002; Chapin et al., 1999; Fetz, 1999; Serruya et al., 2002; Taylor et al., 2002; Carmena et al., 2003; Andersen et al., 2004; Wolpaw and McFarland, 1994, 2004; Leuthardt et al., 2004; Hochberg et al., 2006; Schalk et al., 2008; Velliste et al., 2008; Ganguly and Carmena, 2009). Three-dimensional (3-D) control has been reported only for intracortical signals (i.e., neuronal action potentials) in monkeys (Taylor et al., 2002; Velliste et al., 2008).

Both actual movement and movement imagery are accompanied by changes in the amplitudes of certain EEG rhythms, specifically 8–12-Hz mu rhythms and 18–30-Hz beta rhythms. These changes are focused over sensorimotor cortex (Pfurtscheller et al, 2008) in a manner consistent with the homuncular organization of this cortical region (Woolsey, 1958). Thus, in our earlier demonstrations that people could learn to use EEG for two-dimensional (2-D) movement control, we began the training process by using for control the mu- and/or beta-rhythms changes normally associated with left-hand or right-hand movement imagery (Wolpaw and McFarland, 1994 and 2004). In the present study, we extended this strategy to 3-D control by beginning the training process from the mu- and/or beta-rhythm changes normally associated with left-hand, right-hand, or foot movement imagery (Morash et al, 2008; Pfurtscheller et al, 2008). The results show that EEG can support 3-D movement control. Thus, they may accelerate the development of BCIs useful to people with severe motor disabilities.

2. Methods

The methodology is summarized here. Additional detail is available elsewhere (Wolpaw and McFarland, 2004; McFarland et al., 2006). The BCI users were four adults, one woman and three men, aged 29–59. One man (User 2) had a spinal cord injury (T7) and was confined to a wheelchair. Three had participated in earlier BCI studies (e.g., McFarland et al., 2008), and one had no previous BCI experience. All gave informed consent for the study, which was reviewed and approved by the New York State Department of Health Institutional Review Board.

The BCI user sat in a reclining chair (or in his own wheelchair) facing a video screen and remained motionless. BCI operation and data collection were supported by the general-purpose BCI software platform BCI2000 (Schalk et al., 2004) in conjunction with a 64-channel SA Instrumentation amplifier and a Data Translation DT-3003 64 channel A/D board. EEG was recorded from 64 scalp locations (Sharbrough et al., 1991) by 9-mm tin electrodes embedded in a cap (Electro-cap International) and referenced to an electrode on the right ear, and was digitized at 160 Hz and stored for later analysis. Each user completed 2–3 sessions/wk. A session comprised eight 3-min runs separated by 1-min breaks and each run averaged 14–25 trials.

For the first 1–4 sessions (i.e., 8–32 3-min runs totaling 24–96 min), the users practiced 1-D control in each dimension of movement (i.e., vertical, horizontal, and depth dimensions; 2–3 runs/session for each dimension). For the next 10–12 sessions (i.e., 80–96 runs totaling 4–5 hr), they practiced 2-D control with the three possible pairings of the three dimensions (i.e., 2–3 runs/session with each pairing). Finally, the users moved on to full 3-D control, and each completed 21–42 sessions (i.e., 168–336 runs totaling 8–17 hr) on the standard 3-D task (Fig. 1). Performance on the 3-D task improved steadily as each user gradually gained better control over the EEG features that controlled cursor movement, and as the iterative adaptive feature selection and weighting procedures (see below) progressively modified the set of EEG features used and adjusted their weights so as to vest control of cursor movement in those features that the user was best able to control. Training was continued until the progressive improvement over sessions was no longer clearly apparent (i.e., until performance began to asymptote).

Figure 1.

Figure 1

Open in a new tab

The 3-D movement control format. The large screen image on the left shows the virtual 3-D cube with the eight possible targets in the corners and the cursor in the center. The smaller screen images show the sequence of steps in one trial: (1) a target appears; (2) 1 s later the cursor appears and moves in three dimensions controlled by the user’s EEG activity as described in the text; (3) the cursor reaches the target; (4) the target turns yellow for 1.5 s; (5) the screen is blank for 1 s and then the next trial begins. (Step 2 lasts up to 15 s. If the cursor does not reach the target in this time, the screen goes blank for 1.5 s prior to step 5.)

To control each dimension of cursor movement (horizontal, vertical, and depth), the digitized data from three or four electrodes over sensorimotor cortex of both hemispheres were re-referenced according to a large Laplacian transform (McFarland et al., 1997). Every 50 ms, the frequency spectrum of the previous 400-ms segment from each electrode was computed by a 16th-order autoregressive algorithm (McFarland and Wolpaw, 2008). The logarithms of the amplitudes in specific 3-Hz-wide frequency bands (center frequencies 10–31 Hz) were the EEG features that controlled cursor movements. One or more of these features comprised the control signal (i.e., the independent variable) in a linear equation that specified cursor movement in a particular dimension (McFarland et al., 2006). That is, if ΔV was vertical cursor movement, Sv was the control signal for vertical movement, bv was the gain, and av was the mean value of Sv for the user’s previous performance,

ΔV=bv(Sv−av) (1)

was the function that determined each vertical cursor movement. Similarly, if ΔH was the horizontal cursor movement,

ΔH=bh(Sh−ah) (2)

was the function that determined horizontal cursor movement. Finally, if ΔD was the cursor movement in depth,

ΔD=bd(Sd−ad) (3)

was the function that determined cursor movement in depth. Movements in each dimension occurred 20 times/s.

Initial feature selection was based on a preliminary screening in which the user imagined specific limb movements (Wolpaw and McFarland, 1994). The user was asked to imagine left-hand, right-hand, or foot movement. The features selected initially and at each of the periodic reevaluations came from electrodes located over sensorimotor cortex. Feature selection was then periodically updated between sessions by a stepwise multiple regression procedure (SAS Institute Inc) (Wolpaw and McFarland, 2004). Starting with no initial model terms, the feature that most reduced the residual variance (i.e., the variance not accounted for by target location), and did so with p<0.01, was added to the model. Additional features were then added in the same fashion. After each new addition, a backward stepwise regression removed any variables for which p was >0.01. This process continued until no further features satisfied the addition/removal criteria.

The weights assigned to the selected features were determined by least-squares criteria according to the equation:

(X'X)b=X'Y (4)

where X was an m by n matrix formed from the n observations of m features and Y was the vector of n values (i.e., target locations). Solving for b, the vector of feature weights, gave:

b=(X'X)−1X'Y (5)

Following each of these periodic stepwise regression analyses, the features selected for each of the three linear equations were weighted according to its results. Then, after each trial, the weights assigned to these features were automatically adjusted by the LMS algorithm (Haykin, 1996) so as to optimize for each dimension the correlation between target location and cursor position. This continual automatic adaptation used past performance to optimize the feature weights (Wolpaw and McFarland, 2004).

For the online update of feature weights, a prediction error was computed for each control signal at the end of each trial:

et=pt−ot (6)

where pt is the predicted target position on the current trial based on the current values of the features and ot is the actual target position on the dimension in question. Then the weights were updated according to:

wit=etrwit−1 (7)

where wit is the weight at the end of the current trial for the ith feature and r is a constant that determines the rate of adaptation.

The objective of this feature selection and weighting process was to minimize, for each dimension, the squared difference between the actual target position and the cursor position predicted by the EEG control signals. Both the stepwise feature selection and online adaptive algorithms used these same criteria. In summary, to the extent that past performance predicted future performance, these procedures served to optimize for each dimension the correlation between target location and cursor movement (Wolpaw and McFarland, 2004).

3. Results

Table 1 shows, for each user’s final training sessions, the set of EEG features (specific frequency bands from specific scalp electrodes) that comprised the independent variable for each of the three linear equations that controlled the three movement dimensions. These feature sets were the result of the iterative adaptive interactions over the course of training between each user’s control capacities and the feature selection and weighting procedures described In Methods.

Table 1.

The EEG features (specific frequency bands at specific scalp electrodes) that controlled vertical, horizontal, and depth cursor movements for each user. For each feature, the scalp electrode (Sharbrough et al., 1991) and the center frequency of the 3-Hz wide frequency band (in parenthesis) are given.

ELECTRODE (FREQ)
USER
Vert Horiz Depth
1 Cz(26) C4(26) C3(26)
CPz(26)
2 Cz(24) C3(12) C4(12)
C4(12)
3 Cz(19) C3(10) C3(10)
Cz(31) C4(10)
4 Cz(20) C3(23) C3(20)
CPz(20) C4(23) C3(23)
CPz(29)
Open in a new tab

As previously noted, each user completed 21–42 3-D sessions, and 3-D control gradually improved over these sessions. Figure 2 shows this gradual improvement (as percent of trials completed within 7 sec) for each user. The data from the first 21 sessions, for which there were data from all four users, were evaluated by ANOVA. The effect of sessions was significant (F= 3.84, p<0.0001), and thus confirmed that performance improved with continued practice. Performance improved as each user gradually gained better control over the EEG features that controlled cursor movement, and as the iterative adaptive feature selection and weighting procedures (see Methods) periodically modified the set of EEG features used and adjusted their weights so as to vest control of cursor movement in those features that the user could best control. Figure 3 illustrates these progressive changes in user control and in the features used for control with data from one user at early, middle, and late stages of training.

Figure 2.

Figure 2

Open in a new tab

Percent of trials completed (i.e., target reached within 15 sec) for each user as a function of sessions. User A is represented by the blue line, user B by the black line, user C by the green, and user D by the red. Note each user’s gradual improvement over sessions.

Figure 3.

Figure 3

Open in a new tab

Topographies for User 1 at the beginning (sessions 1–3), middle (sessions 10–12), and end (sessions 19–21) of 3-D training, for the correlations at each of the 64 electrodes between the spectral amplitude of the EEG and each dimension of target location. For each dimension of target location, the topography is for the 3-Hz frequency band centered at 26 Hz that provided that dimension's online control signal (i.e., Table 1). (The correlations are shown as R rather than R2 to distinguish negative and positive correlations.) “X” indicates the locations of the electrodes that provided the frequency-band amplitudes that were used online. Note the changes over time in the topographies and magnitudes of control and in the electrodes used for control. The progressive improvement in performance summarized in Figure 2 is accounted for by the increases in the user’s control in the horizontal and depth dimensions, together with the adaptive algorithm’s modifications in the electrodes used for control in the vertical and depth dimensions.

Using three consecutive 3-D sessions at the end of training (336–608 trials from each user), we assessed EEG control and the cursor movement control that it provided. We assessed EEG control by spectral and topographical analyses of the correlations of the average values for each trial of the vertical, horizontal, and depth control signals with the vertical, horizontal, and depth locations of the target (Wolpaw and McFarland, 1994 and 2004). Table 2 gives, for each user, the correlation of each dimension’s control signal with each of the three dimensions of target position. As Table 2 shows, each control signal correlated strongly with its own dimension of target location and showed no or much lower correlations with the other two signals’ dimensions of target location. Thus, users developed three independent control signals: one for each movement dimension.

Table 2.

Correlations (as R2) of the vertical, horizontal, and depth control signals (SV, SH, SD) with their appropriate (highlighted) and inappropriate dimensions of target location (V, H, D). Each control signal is correlated exclusively or most strongly with its own dimension of target location.

User Vertical Signal (SV) Horizontal Signal
(SH)		 Depth Signal (SD)
SV-V SV-H SV-D SH-H SH-V SH-D SD-D SD-V SD-H
1 0.39 0.01 0.00 0.50 0.00 0.00 0.57 0.00 0.00
2 0.29 0.01 0.14 0.37 0.00 0.16 0.16 0.00 0.00
3 0.37 0.03 0.28 0.47 0.09 0.04 0.15 0.04 0.01
4 0.20 0.00 0.03 0.09 0.00 0.01 0.09 0.00 0.00
Open in a new tab

Across the four users, performance did not correlate with amount of 3-D training. User 1, who achieved the best control (Table 2), had the least number of 3-D training sessions (21 sessions, or 8.4 hrs). User 4, who achieved the least control, had 29 3-D sessions (11.6 hrs), while Users 2 and 3 had 26 and 42 (10.4 and 16.8 hrs), respectively. (User 2 was the person with a spinal cord injury.) Nor did performance correlate clearly with total amount of BCI training (i.e., including participation in previous studies). Users 1–4 had total BCI experience of 102, 367, 115, and 57 hrs., respectively. While the least successful user had the least experience, the most successful user had less experience than Users 2 or 3. The differences in performance evident in Table 2 may reflect inter-user differences in the prominence of sensorimotor rhythms in the scalp-recorded EEG (i.e., differences in signal-to-noise ratio) and/or in the effectiveness of the interactive adaptations in the algorithm and the brain over the course of training. Further improvements of the user-specific adaptations in the feature extraction and translation procedures might enable more users to achieve good performance.

To determine whether the users were controlling the three dimensions simultaneously or sequentially (e.g., moving up, then right, then forward), we evaluated for the first 0.5 s of each trial the individual movements in each of the three dimensions (which occurred every 50 ms) to determine whether a correct movement (i.e., toward the target) in one dimension affected the probability that the simultaneous movements in the other two dimensions were correct. If the users were controlling the dimensions sequentially (i.e., one at a time), the probability that a correct movement in one dimension was accompanied by correct movements in the other two dimensions would be less than that predicted by simply multiplying the fractions of all vertical, horizontal, and depth movements that were correct together. This was not the case: the actual probabilities of simultaneous correct movements were 145%, 164%, 151%, and 181% of their expected values for Users 1–4, respectively. The fact that these probabilities were greater than 100% of expectation indicates that the users were not simply controlling one dimension at a time; they were controlling the three dimensions simultaneously.

Figure 4 shows, for each user, the topographies (nose at top) of the correlations for each of the 64 electrodes between the spectral amplitude of the EEG and each dimension of target location. For each dimension of target location, the topography is for the frequency band that made the largest contribution to that dimension's online control signal. “X” indicates the locations of the electrodes that provided the frequency-band amplitudes that were used online. Table 1 gives the frequencies of the topographies. While the correlations are all focused over sensorimotor cortex, they differ markedly across users as a result of the iterative adaptive interactions during training between the brain and the feature selection and weighting procedures.

Figure 4.

Figure 4

Open in a new tab

Topographies (nose at top) for each user (1–4) of the correlations for each of the 64 electrodes between the spectral amplitude of the EEG and each dimension of target location. For each dimension of target location, the topography is for the frequency band that made the largest contribution to that dimension's online control signal. (The correlations are shown as R rather than R2 to distinguish negative and positive correlations.) “X” indicates the locations of the electrodes that provided the frequency-band amplitudes that were used online. The center frequencies of the 3-Hz frequency bands of each user’s topographies are given in Table 1. While the correlations are all focused over sensorimotor cortex, they differ markedly across users as a result of inter-user differences in the course of the iterative adaptive interaction between user and system that occurs during training (see Methods). For example, User 1 controlled the three movement dimensions with 26-Hz activity from three different scalp electrodes, while User 3 controlled vertical movement with the left-right difference in 10-Hz activity, horizontal movement with 19- and 31-Hz activity at the vertex, and depth movement with 10-Hz activity on the left.

Figure 5 shows User 1’s control in more detail. Figure 5A shows the topographies of the correlations of 26-Hz activity (Table 1) with the three dimensions, with the electrode(s) that controlled each dimension marked. Figure 5B shows the spectra for the correlations (as R2) between activity at the electrode that made the largest (or only) contribution to the control signal for that dimension and the target location in that dimension. Each feature correlates strongly with its appropriate dimension of target location and not with the other dimensions. Figure 5C shows single EEG traces from electrodes used in the vertical, horizontal, and depth control signals for trials in which the target was at the top or bottom, right or left, or back or front of the cube, respectively (Fig. 1). They illustrate the strong EEG feature control the user employed to move the cursor to the target.

Figure 5.

Figure 5

Open in a new tab

Topographical and spectral properties of EEG control for User 1. In this user, movement in each dimension was controlled by 26-Hz activity from specific scalp electrodes (Table 1).

A: Scalp topographies (nose at top) of the correlations of the 26-Hz frequency band with the vertical, horizontal, and depth target locations, respectively. The electrode(s) that controlled each dimension of movement are marked. (The correlations are shown as R rather than R2 in order to distinguish negative and positive correlations.)

B: Spectra for the correlations (shown as R2) of the activity at the scalp electrode that made the largest (or only) contribution to the control signal for each dimension of cursor movement with the three dimensions of target location. The correlations with the vertical, horizontal, and depth dimensions are red, blue, and black lines, respectively. It is clear that activity at the electrode that provided each control signal correlated strongly with its appropriate dimension of target location and did not correlate with the other dimensions. Furthermore, the correlation was focused in the appropriate (i.e., in this case, 26-Hz) frequency band.

C: Samples of EEG activity from single trials. The traces are single 400-msec epochs of Laplacian-derived EEG from one electrode. On the left are traces from scalp electrode CPz (the major source of the vertical control signal) for trials in which the target was at the top or bottom of the cube. In the middle are traces from electrode C4 (the source of the horizontal control signal) for trials in which the target was on the right or left side of the cube. On the right are traces from electrode C3 (the source of the depth control signal) for trials in which the target was at the front or back of the cube. They illustrate the strong 26-Hz control that the user employed to move the cursor to the target.

The EEG control summarized in Table 2 and illustrated in Figures 4 and 5 gave each user 3-D movement control. Users 1–4 reached the target within the time allowed in 93%, 78%, 76%, and 56% of the trials, respectively, and their median movement times for these completed trials were 1.6, 2.9, 3.2, and 4.9 s, respectively. Furthermore, the first of the eight possible target locations reached by the cursor correlated strongly with the actual target location (P<0.0001 by χ2 for each user), indicating that movement was not random. Supplementary Videos 1 and 3 show the average trajectory to each target for Users 1 and 3, respectively. Supplementary Videos 2 and 4 show real-time performance for Users 1 and 3, respectively. While both users have 3-D control (i.e., Table 2), User 1’s performance is clearly superior.

An ancillary study started the cursor from locations other than the center of the cube (i.e., near the corner opposite the target). In spite of the variable starting points and the greater distance to the target, the percent of targets reached did not differ significantly from the percent reached with the standard (center start) format (p>0.25 by ANOVA).

We also evaluated forearm and calf EMG activity during performance. EMG activity was usually well below 10% of MVC throughout and uncorrelated with target location. In the few instances in which EMG did correlate with target location in a particular dimension, the correlation of the EEG control signal with target location remained significant after the effect of the EMG correlation was removed. The results confirmed previous data (e.g., Vaughan et al., 1998; Wolpaw and McFarland, 2004) indicating that EEG-based cursor control does not depend on covert contractions of muscle groups strongly represented in the sensorimotor cortex areas that produce the EEG features used for control.

4. Discussion

The results show that people can learn to use scalp-recorded EEG activity to control three movement dimensions simultaneously and independently (Table 2). This control develops through training as the user gradually acquires better control of the EEG features that control movement, and as the BCI system gradually focuses on those features that the user can best control. Thus, 3-D control is basically a skill (i.e., an ability acquired through practice (Brown, 1993)) that user and system master together. Users did not have greater difficulty acquiring control of the second or third dimension than they did acquiring control of the first dimension. They did need further practice to control three dimensions simultaneously. As training proceeded and performance improved, users reported that the motor imagery they initially employed became less important and cursor control became more automatic. In this characteristic, the skill of EEG-based 3-D movement control resembles conventional motor skills, in which training leads to automaticity and to performance that is less dependent on attention (Moors and de Houwer, 2006).

Given this understanding of 3-D control as a skill acquired through practice, appropriate modifications in the training protocol might be expected to facilitate learning and improve final performance. For example, Table 2 indicates that several users were less successful in controlling cursor movement in the depth dimension than in the horizontal and vertical dimensions. This difference could be due in part to the fact that the depth dimension was less salient in the display that we used (i.e., Fig. 1, Videos 2 and 4). Thus, adding binocular cues for depth (e.g., through a stereoscopic display) might facilitate the user’s acquisition of depth control.

This demonstration of EEG-based 3-D movement control has practical implications. It suggests that, with further development, EEG-based BCI systems might prove capable of restoring movement control to people who are paralyzed.

The factors important for movement control

The factors that determine the control possible with EEG, ECoG, and intracortical BCI methods are as yet unknown. Many efforts to develop BCI multidimensional movement control have started from the assumption that high-resolution signals are essential, and thus they have used intracortical electrodes to record neuronal action potentials or local field potentials (Serruya et al., 2002; Taylor et al., 2002; Carmena et al., 2003, Ganguly and Carmena, 2009; Andersen et al., 2004; Hochberg et al., 2006; Velliste et al., 2008). Intracortical methods provide the highest resolution signals. At the same time, they face difficulties in ensuring stable long-term function (Otto et al., 2006), and the information available from neurons may have inherent limitations (Rokni et al., 2007). Furthermore, by demonstrating EEG-based 3-D control in humans, the present study adds to recent evidence suggesting that signal resolution may not be the critical limiting factor.

Figure 6 shows the distributions of target-acquisition times for two studies of center-out 2-D control in humans, one using a cortical neuron-based BCI (Hochberg et al., 2006) and one using an EEG-based BCI (Wolpaw and McFarland, 2004). The figure also includes the distribution of times for conventional muscle-based joystick control. The two BCIs studies had similar protocols, and they yielded nearly identical distributions of target-acquisition times. Both are slower and far less consistent than joystick control. Even after the considerable BCI training provided in each study, performance remains inconsistent. Such inconsistency is typical of BCI studies (e.g., compare supplementary videos 1 and 8 of Hochberg et al. (2006)), including the present one. Indeed, the most striking feature of the comparison in Figure 6 is the close similarity of the two BCI distributions. This similarity is remarkable, given that one BCI used single-neuron activity recorded within the cortex while the other used EEG recorded from the scalp. It suggests that their inconsistency was not related to signal resolution (which was high for the neuronal BCI and low for the EEG BCI), but rather reflects another factor that similarly limits both high-resolution and low-resolution BCI methods.

Figure 6.

Figure 6

Open in a new tab

Distributions of target-acquisition times (i.e., time from target appearance to target hit) on a 2-D center-out cursor-movement task for joystick control (black), EEG-based BCI control (blue), and cortical neuron-based BCI control (red). The EEG-based and neuron-based BCIs perform similarly, and both are slower than and much less consistent than the joystick. For both BCIs in a substantial number of trials, the target is not reached even in the 7 s allowed. Such inconsistent performance is typical of movement control by present-day BCIs, regardless of what brain signals they use. (The joystick data and neuron-based BCI data are from Hochberg et al. (2006). The EEG-based BCI data are from Wolpaw and McFarland (2004).)

Movement control has been traditionally viewed as highly localized (e.g., Woolsey, 1958). However, recent work indicates that movements are controlled by distributed cortical networks that include not only primary motor cortex, but other areas as well (e.g., premotor, prefrontal) (Dum and Strick, 2002 and 2005; Aflalo and Graziano, 2006; Meier et al., 2008; Ledberg et al., 2006). These networks appear to function through synchronous oscillations in their constituent parts (Bullmore and Sporns, 2009; Salinas and Sejnowski, 2001; Sejnowski and Paulsen, 2006; Zhang et al., 2008). This new understanding suggests that present-day BCI movement control may be limited and inconsistent (e.g., Fig. 6) in large part because it relies only on signals from a single cortical area. Neuron-based control has typically focused on neurons from a few cubic mm of cortex, and has begun by using the neuronal activity observed during actual movements (e.g., Andersen et al., 2004; Hochberg et al, 2006; Velliste et al., 2008). Similarly, EEG-based control has focused on rhythms recorded over sensorimotor cortex, and has begun by using the rhythm changes observed during movement imagery (e.g., McFarland et al., 2008; Wolpaw and McFarland, 1994 and 2004).

This new understanding of the highly distributed nature of motor control suggests that the performance of BCIs, whether they use EEG, ECoG, or intracortical signals, might possibly be improved by extracting signal features from multiple cortical areas and using adaptive algorithms similar to that of the present study to combine them to control movements. Employing signals from multiple areas might allow BCI operation to more closely mimic normal neuromuscular movement control. We speculate that, by eliminating the limit that may be imposed on BCIs that use only one area, this approach might allow the control capacities of the different signal types to be more fully realized and could produce more reliable performance. Furthermore, identification of the combinations of areas that perform best could illuminate the interactions most critical for movement control.

Development of BCIs to restore movement control

The primary goal of BCI research is to restore communication and control to people with severe motor disabilities. This study advances the goal in two ways. First, its demonstration of EEG-based 3-D control suggests that BCIs capable of controlling neuroprostheses, robotic arms, or comparable devices might not require surgical implantation of recording electrodes. This work complements and extends the recent demonstration that an EEG-based BCI can control a sequential movement-selection (i.e., reach- and- grasp) action (McFarland et al., 2008).

Second, the present demonstration that EEG can support 3-D movement control should facilitate evaluation of the hypothesis that consistency can be improved by using signals from multiple cortical areas. The inconsistency of the movement control provided by current BCIs, regardless of which signals they use, is probably the single greatest impediment to their practical use. Unless and until BCI-based movement control becomes consistent, it will remain a laboratory curiosity, with little value to people who need to operate neuroprostheses or robotic devices in their daily lives. Because EEG is noninvasive, BCI research studies can readily record EEG from multiple areas and explore the usefulness of their various combinations. Furthermore, by doing this they might also suggest promising combinations of areas for exploration with high-resolution invasive methods. By demonstrating the potential of EEG-based BCIs, the present results show that EEG is a valid avenue for studying the value of combining signals from different areas, and they thereby encourage such studies.

Efforts to realize the clinical potential of EEG-based movement control must also address other important issues. These include the achievement of continuous sequential control, such as the ability to move to a location, to stay there while performing another action, to immediately move to another location, etc. Also important for practical applications will be development of improved training methods and better adaptive algorithms to enable most users to attain, and maintain, good control after relatively brief training. As hypothesized above, algorithms that use features from additional brain areas and/or measures of cross-channel relationships might possibly provide more reliable control (Varela et al., 2001). Such expanded algorithms will need to incorporate better methods for identifying the most useful signal features from among a large feature set on the basis of relatively limited bodies of data (i.e., better regularization methods). In real-time BCI applications, the data most useful for selecting signal features are those from very recent performance, and these data are necessarily limited in amount. With such limited data, an algorithm that lacks effective regularization tends to overfit the data from a user’s recent performance and thus provides parameter values that are not appropriate for future performance (i.e., they do not generalize well). While the present study used stepwise regression methods to select features, alternative procedures (e.g., Tibshirani, 1996; Farquhar, 2009; van Gerven, 2009) might prove more effective.

5. Conclusions

By demonstrating that an EEG-based BCI can support 3-D movement, this study shows that high signal resolution is not essential for complex movement control and suggests that other factors are more important. We hypothesize that combinations of signals from multiple cortical areas might produce more consistent performance. With further development, it may eventually become possible for people with severe neuromuscular disorders to operate devices such as a robotic arm, a motorized wheelchair, or a neuroprosthesis with brain signals recorded from the scalp.

Supplementary Material

Supplementary video 1. Supplementary Video 1.

Average cursor movement trajectories for User 1. The virtual cubic workspace rotates to enable all eight trajectories to be seen. The pink and yellow targets are the upper-left-front target and the upper-right-front target, respectively. In order to best reveal the movement path, the average includes all trials in which the cursor reached the target within 5 s (i.e., the fastest 88% of the user’s target hits), and the final cursor position is normalized to the center of the target. (Supplementary Video 2 shows this user’s real-time performance.)

To watch Video 1, please click on this link: http://www.bciresearch.org/html/3Dvideo1.html

Download video file (6MB, wmv)
Supplementary video 2. Supplementary Video 2.

In this real-time video, User 1 uses scalp-recorded sensorimotor rhythms (SMRs) to control cursor movement in three dimensions. In each trial, a target appears in one of the eight corners of the three-dimensional cubic workspace. One second later, the cursor appears in the center of the cube and begins to move. As described in the text, cursor movement in each dimension is controlled by SMR activity recorded from the scalp over sensorimotor cortex. The person has learned simultaneous independent control of different SMR features (i.e., for the vertical, horizontal, and depth dimensions, respectively) (see Fig. 3). With this control, the user moves the cursor to the target rapidly and accurately.

To watch Video 2, please click on this link: http://www.bciresearch.org/html/3Dvideo2.html

Download video file (8.8MB, wmv)
Supplementary video 3. Supplementary Video 3.

Average cursor movement trajectories for User 3. The virtual cubic workspace rotates to enable all eight trajectories to be seen. The pink and yellow targets are the upper-left-front target and the upper-right-front target, respectively. In order to best reveal the movement path, the average includes all trials in which the cursor reached the target within 8 s (i.e., the fastest 82% of the user’s target hits), and the final cursor position is normalized to the center of the target. (Supplementary Video 4 shows this user’s real-time performance.)

To watch Video 3, please click on this link: http://www.bciresearch.org/html/3Dvideo3.html

Download video file (6.5MB, wmv)
Supplementary video 4. Supplementary Video 4.

In this real-time video, User 3 uses scalp-recorded sensorimotor rhythms (SMRs) to control cursor movement in three dimensions. In each trial, a target appears in one of the eight corners of the three-dimensional cubic workspace. One second later, the cursor appears in the center of the cube and begins to move. As described in the text, cursor movement in each dimension is controlled by SMR activity recorded from the scalp over sensorimotor cortex. While this person has learned simultaneous independent control of different SMR features (i.e., for the vertical, horizontal, and depth dimensions, respectively (see Table 2)), the movement times vary widely, and in one trial the target is not reached in the 15 s allowed. This user’s control is not as fast or reliable as that of User 1 (see Supplementary Video 2).

To watch Video 4, please click on this link: http://www.bciresearch.org/html/3Dvideo4.html

Download video file (5MB, wmv)

Acknowledgements

This work was supported by grants from NIH (HD30146 (NCMRR, NICHD) and EB00856 (NIBIB & NINDS)) and the James S. McDonnell Foundation. We thank Theresa M. Vaughan and Gerwin Schalk for valuable advice throughout this work and Jonathan S. Carp, Eric W. Sellers, Elizabeth Winter Wolpaw, Chadwick B. J. Boulay, Brandon LaPallo, Scott Brainard, and Peter Brunner for their comments on the manuscript.

References

  1. Aflalo TN, Graziano MS. Possible origins of the complex topographic organization of motor cortex: reduction of a multidimensional space onto a two-dimensional array. Journal of Neuroscience. 2006;26:6288–6297. doi: 10.1523/JNEUROSCI.0768-06.2006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Andersen RA, Musallam S, Pesaran B. Selecting signals for a brain-machine interface. Curr. Opin. Neurobio. 2004;14:720–726. doi: 10.1016/j.conb.2004.10.005. [DOI] [PubMed] [Google Scholar]
  3. Brown L, editor. The New Shorter Oxford English Dictionary. Vol. 2. Oxford: Clarenden Press; 1993. p. 2882. [Google Scholar]
  4. Bullmore E, Sporns O. Complex brain networks: graph theoretical analysis of structural and functional systems. Nature Reviews: Neuroscience. 2009;10:186–198. doi: 10.1038/nrn2575. [DOI] [PubMed] [Google Scholar]
  5. Carmena JM, Lebedev MA, Crist RE, O’Doherty JE, Santucci DM, Dimitrov DF, Patil PG, Henriquez CS, Nicolelis MAL. Learning to control a brain-machine interface for reaching and grasping by primates. PloS Biol. 2003;1:1–16. doi: 10.1371/journal.pbio.0000042. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Chapin JK, Moxon KA, Markowitz RS, Nicolelis MAL. Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nat. Neurosci. 1999;2:664–670. doi: 10.1038/10223. [DOI] [PubMed] [Google Scholar]
  7. Dum RP, Strick PL. Motor areas in the frontal lobe of the primate. Physiology and Behavior. 2002;77:677–682. doi: 10.1016/s0031-9384(02)00929-0. [DOI] [PubMed] [Google Scholar]
  8. Dum RP, Strick PL. Frontal lobe inputs to the digit representations of the motor areas on the lateral surface of the hemisphere. Journal of Neuroscience. 2005;25:1375–1386. doi: 10.1523/JNEUROSCI.3902-04.2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Farquhar J. A linear feature space for simultaneously learning of spatio-spectral filters in BCI. Neural Networks. 2009;22:1278–1285. doi: 10.1016/j.neunet.2009.06.035. [DOI] [PubMed] [Google Scholar]
  10. Fetz EE. Real-time control of a robotic arm. Nat. Neurosci. 1999;2:583–584. doi: 10.1038/10131. [DOI] [PubMed] [Google Scholar]
  11. Ganguly K, Carmena JM. Emergence of a stable cortical map for neuroprosthetic control. PLoS Biology. 2009;7:e1000153. doi: 10.1371/journal.pbio.1000153. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Haykin S. Adaptive Filter Theory. Upper Saddle River, NJ: Prentice-Hall; 1996. [Google Scholar]
  13. Hochberg LR, Serruya MD, Friehs GM, Mukand JA, Saleh M, Caplan AH, Branner A, Penn DRD, Donoghue JP. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature. 2006;442:164–171. doi: 10.1038/nature04970. [DOI] [PubMed] [Google Scholar]
  14. Ledberg A, Bressler SL, Ding M, Coppola R, Nakamura R. Large-scale visuomotor integration in the cerebral cortex. Cerebral Cortex. 2007;17:44–62. doi: 10.1093/cercor/bhj123. [DOI] [PubMed] [Google Scholar]
  15. Leuthardt EC, Schalk G, Wolpaw JR, Ojemann JG, Moran DW. A brain-computer interface using electrocorticographic signals in humans. J. Neural. Eng. 2004;1:63–71. doi: 10.1088/1741-2560/1/2/001. [DOI] [PubMed] [Google Scholar]
  16. McFarland DJ, Krusienski DJ, Sarnacki WA, Wolpaw JR. Emulation of computer mouse control with a noninvasive brain-computer interface. J. Neural Eng. 2008;5:101–110. doi: 10.1088/1741-2560/5/2/001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. McFarland DJ, Krusienski DJ, Wolpaw JR. Brain-computer interface signal processing at the Wadsworth Center: mu and sensorimotor beta rhythms. Prog. Brain Res. 2006;159:411–419. doi: 10.1016/S0079-6123(06)59026-0. [DOI] [PubMed] [Google Scholar]
  18. McFarland DJ, McCane LM, David SV, Wolpaw JR. Spatial filter selection for EEG-based communication. EEG Clin Neurophysiol. 1997;103:386–394. doi: 10.1016/s0013-4694(97)00022-2. [DOI] [PubMed] [Google Scholar]
  19. McFarland DJ, Wolpaw JR. Sensorimotor rhythm-based brain-computer interface (BCI): model order selection for autoregressive spectral analysis. J. Neural Eng. 2008;5:155–162. doi: 10.1088/1741-2560/5/2/006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Meier JD, Afalo TN, Kastner S, Graziano MS. Complex organization of human primary motor cortex: a high-resolution fMRI study. Journal of Neurophysiology. 2008;100:1800–1812. doi: 10.1152/jn.90531.2008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Moors A, De Houwer J. Automaticity: A theoritical and conceptual analysis. Psychol. Bull. 2006;132:297–302. doi: 10.1037/0033-2909.132.2.297. [DOI] [PubMed] [Google Scholar]
  22. Morash V, Bai O, Furlani S, Lin P, Hallett M. Classifying EEG signals preceding right hand,m left hand, tongue, and right foot movements and motor imagery. Clin. Neurophysiol. 2008;119:2570–2578. doi: 10.1016/j.clinph.2008.08.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Otto KJ, Johnson MD, Kipke DR. Voltage pulses change neural interface properties and improve unit recordings with chronically implanted microelectrodes. IEEE Trans. Biomed. Eng. 2008;53:333–340. doi: 10.1109/TBME.2005.862530. [DOI] [PubMed] [Google Scholar]
  24. Pfurtscheller G, Scherer R, Muller-Putz GR, Lopes da Silva FH. Short-lived brain state after cued motor imagery in naive subjects. Eur J Neurosci. 2008;28:1419–1426. doi: 10.1111/j.1460-9568.2008.06441.x. [DOI] [PubMed] [Google Scholar]
  25. Rokni U, Richardson AG, Bizzi E, Seung HS. Motor learning with unstable neural representations. Neuron. 2007;54:653–666. doi: 10.1016/j.neuron.2007.04.030. [DOI] [PubMed] [Google Scholar]
  26. Salinas E, Sejnowski TJ. Correlated neuronal activity and the flow of neural information. Nature Reviews: Neuroscience. 2001;2:539–550. doi: 10.1038/35086012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Schalk G, McFarland DJ, Hinterberger T, Birbaumer N, Wolpaw JR. BCI2000: A general-purpose Brain-Computer interface (BCI) system. IEEE Trans. Biomed. Eng. 2004;51:1034–1043. doi: 10.1109/TBME.2004.827072. [DOI] [PubMed] [Google Scholar]
  28. Schalk G, Miller KJ, Anderson NR, Wilson JA, Smyth MD, Ojemann JG, Moran DW, Wolpaw JR, Leuthardt EC. Two-dimensional movement control using electrocorticographic signals in humans. J. Neural Eng. 2008;5:75–84. doi: 10.1088/1741-2560/5/1/008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Sejnowski TJ, Paulsen O. Network oscillations: emerging computational principals. Journal of Neuroscience. 2006;26:1673–1776. doi: 10.1523/JNEUROSCI.3737-05d.2006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Serruya MD, Hatsopoulos NG, Paminski L, Fellows MR, Donoghue JP. Instant neural control of a movement signal. Nature. 2002;416:141–142. doi: 10.1038/416141a. [DOI] [PubMed] [Google Scholar]
  31. Sharbrough F, Chatrian CE, Lesser RP, Luders H, Nuwer M, Picton TW. American Electroencephalographic Society guidelines for standard electrode position nomenclature. J. Clin. Neurophysiol. 1991;8:200–202. [PubMed] [Google Scholar]
  32. Taylor DA, Helms Tillery S, Schwartz AB. Direct cortical control of 3D neuroprosthetic devices. Science. 2002;296:1829–1832. doi: 10.1126/science.1070291. [DOI] [PubMed] [Google Scholar]
  33. Tibshirani R. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, series B. 1996;58:267–288. [Google Scholar]
  34. van Gerven M, Hesse C, Jensen O, Heskes T. Interpreting single trial data using groupwise regularization. Neuroimage. 2009;46:665–676. doi: 10.1016/j.neuroimage.2009.02.041. [DOI] [PubMed] [Google Scholar]
  35. Varela F, Lachaux J-P, Rodriguez E, Martinerie J. The brainweb: Phase synchronization and large-scale integration. Nature Reviews: Neuroscience. 2001;2:229–239. doi: 10.1038/35067550. [DOI] [PubMed] [Google Scholar]
  36. Vaughan TM, Miner LA, McFarland DJ, Wolpaw JR. EEG-based communication: analysis of concurrent EMG activity. Electroencephalogr. Clin. Neurophysiol. 1998;107:428–433. doi: 10.1016/s0013-4694(98)00107-2. [DOI] [PubMed] [Google Scholar]
  37. Velliste M, Perel S, Spalding MC, Whitford AS, Schwartz AB. Cortical control of a prosthetic arm for self-feeding. Nature. 2008;453:1098–1101. doi: 10.1038/nature06996. [DOI] [PubMed] [Google Scholar]
  38. Wolpaw JR, Birbaumer N. Brain-Computer Interfaces for Communication and Control. In: Selzer ME, Cohen L, Gage FH, Clarke S, Duncan PW, editors. Textbook for Neural Repair and Repair and Rehabilitation. Cambridge: Cambridge Univ. Press; 2006. pp. 602–614. [Google Scholar]
  39. Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM. Brain-computer interfaces for communication and control. Clin. Neurophysiol. 2002;113:767–791. doi: 10.1016/s1388-2457(02)00057-3. [DOI] [PubMed] [Google Scholar]
  40. Wolpaw JR, McFarland DJ. Multichannel EEG-based brain-computer communication. Electroencephalogr. Clin. Neurophysiol. 1994;90:444–449. doi: 10.1016/0013-4694(94)90135-x. [DOI] [PubMed] [Google Scholar]
  41. Wolpaw JR, McFarland DJ, Neat GW, Forneris CA. Electroencephalogr. Clin. Neurophysiol. 1991;78:252–259. doi: 10.1016/0013-4694(91)90040-b. [DOI] [PubMed] [Google Scholar]
  42. Wolpaw JR, McFarland DJ. Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proc. Natl. Acad. Sci. 2004;101:17849–17854. doi: 10.1073/pnas.0403504101. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Wolpaw JR, McFarland DJ, Neat GW, Forneris CA. An EEG-based brain-computer interface for cursor control. Electroencephalogr. Clin. Neurophysiol. 1991;78:252–259. doi: 10.1016/0013-4694(91)90040-b. [DOI] [PubMed] [Google Scholar]
  44. Woolsey CN. Organization of somatic sensory and motor areas of the cerebral cortex. In: Harlow HF, Woolsey CN, editors. Biological and Biochemical Basis of Behavior. Madison: University of Wisconsin Press; 1958. pp. 63–81. [Google Scholar]
  45. Zhang Y, Wang X, Bressler SL, Chen Y, Ding M. Prestimulus cortical activity is correlated with speed of visuomotor processing. Journal of Cognitive Neuroscience. 2008;20:1915–1925. doi: 10.1162/jocn.2008.20132. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary video 1. Supplementary Video 1.

Average cursor movement trajectories for User 1. The virtual cubic workspace rotates to enable all eight trajectories to be seen. The pink and yellow targets are the upper-left-front target and the upper-right-front target, respectively. In order to best reveal the movement path, the average includes all trials in which the cursor reached the target within 5 s (i.e., the fastest 88% of the user’s target hits), and the final cursor position is normalized to the center of the target. (Supplementary Video 2 shows this user’s real-time performance.)

To watch Video 1, please click on this link: http://www.bciresearch.org/html/3Dvideo1.html

Download video file (6MB, wmv)
Supplementary video 2. Supplementary Video 2.

In this real-time video, User 1 uses scalp-recorded sensorimotor rhythms (SMRs) to control cursor movement in three dimensions. In each trial, a target appears in one of the eight corners of the three-dimensional cubic workspace. One second later, the cursor appears in the center of the cube and begins to move. As described in the text, cursor movement in each dimension is controlled by SMR activity recorded from the scalp over sensorimotor cortex. The person has learned simultaneous independent control of different SMR features (i.e., for the vertical, horizontal, and depth dimensions, respectively) (see Fig. 3). With this control, the user moves the cursor to the target rapidly and accurately.

To watch Video 2, please click on this link: http://www.bciresearch.org/html/3Dvideo2.html

Download video file (8.8MB, wmv)
Supplementary video 3. Supplementary Video 3.

Average cursor movement trajectories for User 3. The virtual cubic workspace rotates to enable all eight trajectories to be seen. The pink and yellow targets are the upper-left-front target and the upper-right-front target, respectively. In order to best reveal the movement path, the average includes all trials in which the cursor reached the target within 8 s (i.e., the fastest 82% of the user’s target hits), and the final cursor position is normalized to the center of the target. (Supplementary Video 4 shows this user’s real-time performance.)

To watch Video 3, please click on this link: http://www.bciresearch.org/html/3Dvideo3.html

Download video file (6.5MB, wmv)
Supplementary video 4. Supplementary Video 4.

In this real-time video, User 3 uses scalp-recorded sensorimotor rhythms (SMRs) to control cursor movement in three dimensions. In each trial, a target appears in one of the eight corners of the three-dimensional cubic workspace. One second later, the cursor appears in the center of the cube and begins to move. As described in the text, cursor movement in each dimension is controlled by SMR activity recorded from the scalp over sensorimotor cortex. While this person has learned simultaneous independent control of different SMR features (i.e., for the vertical, horizontal, and depth dimensions, respectively (see Table 2)), the movement times vary widely, and in one trial the target is not reached in the 15 s allowed. This user’s control is not as fast or reliable as that of User 1 (see Supplementary Video 2).

To watch Video 4, please click on this link: http://www.bciresearch.org/html/3Dvideo4.html

Download video file (5MB, wmv)

RESOURCES