How does Neuralink measure the performance of its interface?

If you’ve been following Neuralink and its first user, Noland Arbaugh, you’ve probably noticed that Noland keeps breaking the brain-computer interface (BCI) performance record. The record value is given in bits per second, which is a unit used mainly in digital information technology, but what exactly does it mean in terms of neural interface speed? How do they measure BCI performance? In this article, we’ll try to answer that question.

Noland’s record

Shortly after his surgery, Noland broke official record for BCI transfer speed. We learned about it during Neuralink’s all hands meeting in March 2024, where Noland with Bliss Chapman presented their achievements with the interface and initial testing results. At that time, the record speed was 4.61 BPS.

In May 2024 blog post Neuralink published a graph of Noland’s maximum daily performances. It showed the transfer rates that Noland was achieving shortly after implantation. There is an obvious decrease in the values ​​during the problem with the electrode threads being pulled out and later an increase again after decoding algorithm changes. The maximum speeds in the graph reach around 8 BPS.

In July 2024, at a presentation on how to solve the problem of loose electrodes, it was reported that Noland had broken his own record again, even with only 15% of active electrodes. This was on the 133rd day after his surgery and the value was 9.51 BPS.

This is the latest information on the max transfer rate that has been achieved. However, we have no information about the values ​​​​achieved by the other two users of N1 Telepathy interface.

Bits or bytes?

There is a certain discrepancy in Neuralink’s information about the interface’s transfer speed – it says BPS, which may indicate that it is “bytes per second”, but it is actually called “bits per second”. So how is it?

Wikipedia defines bit and byte per second as follows:

“The bit is the most basic unit of information in computing and digital communication. The name is a portmanteau of binary digit. The bit represents a logical state with one of two possible values. These values are most commonly represented as either “1” or “0”, but other representations such as true/false, yes/no, on/off, or +/− are also widely used.”

“The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable unit of memory in many computer architectures.”

The difference between a bit and a byte is therefore 8-fold, and the disparity in their notation is the lowercase and uppercase letter “B” (bps vs. Bps). However, Neuralink (for unknown reasons) mostly shows BPS, which complicates the situation a bit more. So is it bits or bytes?

It is most likely bits, because in all presentations, videos and interviews, company representatives always talk about bits, not bytes.

Webgrid

Webgrid is a game that Noland Arbaugh plays every day. It is also played by Neuralink’s experimental monkeys and by individuals with implanted interfaces from other companies and research institutions. It is used to measure the performance of BCIs. Thanks to Neuralink, the game is now available for everyone to play (here).

The following information is displayed on Webgrid page (confirming that the performance is measured in bits, not bytes per second):

The grid has 30×30 fields and beating Noland’s record is not easy at all. The best score in Neuralink, 17.1 BPS on a 35×35 grid, is held by now former employee Bliss Chapman.

Ok, but what does the measured value indicate?

So we already have the metric and its measured value, but we don’t know exactly what this value indicates and how it is measured.

What does it mean that the transmission speed of Noland’s interface reaches almost 10 bits per second?

If we compare this with the connection speeds in computer networks, this is practically zero, negligible speed. The first commonly used Internet modems in the 1990s had 56kb/s throughput, i.e. 5600 times more than the Neuralink interface. This makes it seem that the performance of the interface is extremely low. How much information can be transferred at a speed of 10 bits per second? And how much data does the interface collect from the user’s brain?

What data streams does the N1 interface generate?

In May 2024, Neuralink announced a competition in data compression. The task description included, among other things, two details regarding device throughput:

• The N1 implant collects approximately 200Mbps of data from its 1024 electrodes.
• Wirelessly, using Bluetooth, it can then transmit about 1Mbps to a connected phone or computer.

This means that about 200 times compression is needed here. Or rather, partial compression and partial omission of some data, because 200 times compression is probably unrealistic.

Now when we look at Noland’s performance record of about 10bps, we can see that the N1 wireless interface is capable of transmitting 100,000 times more data to the computer. However, we have no information how much data the interface actually transmits.

For the first users of the interface, we must also remember that:

• The N1 Telepathy interface transmits data in only one direction – from the brain to the computer.
• Noland only has about 15% of the electrodes active – the rest is turned off, so the data flow is even smaller.
• The second user, Alex, has about 400 active electrodes (or about 40%).

In any case, it is clear that the stated transfer speed of the N1 interface (Noland’s record – 9.51 bps) has nothing to do with the real digital data flows that go through the interface. So how does Neuralink measure the performance of the interface?

The methodology of the team around Krishna Shenoy

So where does that 9.51 bps come from? We know this from the paper co-authored by recently deceased neuroscientist Krishna Shenoy, a former Neuralink external advisor and a recognized authority on BCI field.

Shenoy and his colleagues at Stanford University devised a general methodology for measuring and testing the performance of BCIs. He also compiled a record table in this field.

Authors of the methodology state that, according to them, defining and measuring performance metrics is the key to comparing, coordinating, and collaborating to move the development of neural interfaces forward toward their widespread use in medicine. Their research is part of the BrainGate2 initiative, and the presented methodology is applicable to both human and non-human primate studies.

Three types of neural interfaces

In order to compile the proposed methodology, Stanford neuroscientists divide BCIs used for communication into three groups:

1. point-and-click cursor control BCIs:
   • used with a virtual keyboard
   • text is generated by pointing and clicking on individual letters of the keyboard
   • text generation speed is limited to around 40 correctly marked characters per minute (ccpm) – study participants are usually unable to achieve more
   • this is how participants in the Neuralink PRIME study use their interfaces
2. brain-to-text attempted handwriting BCIs:
   • the users imagine in their mind that they are writing the given characters by hand
   • character dictionary is practically unlimited – even Chinese/Japanese characters can be decoded
   • much higher text generation speed – the interface that Krishna Shenoy’s team worked on reached around 85 ccpm
   • this method of communication Neuralink has also tested with Noland
3. interfaces that, in conjunction with a language model with a finite vocabulary size, decode whole words directly from the brain:
   • these are not universal – their language models have a limited vocabulary

Three types of metrics

Each of these interface types has its own specifics. That’s why Shenoy’s team came up with three special methods to measure their performance:

1. cwpm (correct words per minute)
2. ccpm (correct characters per minute)
3. achieved bitrate

CWPM

For interfaces that decode whole words or characters from brain activity, the method of measuring the number of correctly written words per minute, cwpm, is suitable. It uses the following formula:

In the formula:

• T is typing rate in units of correct words per minute (cwpm)
• S_c is the correct number of symbols (keys) transmitted in a minute, including spaces and backspaces (deletes), in some period of time t.
• S_i is the number of incorrect symbols per minute, including spaces and backspaces, in some period of time t
• t is the measurement interval, usually minutes to hours
• an average of 5 characters (including spaces) per word is assumed as this is a typical estimate, but 6 is also sometimes used

This method is not very relevant from the perspective of the N1 Telepathy interface, which is a point-and-click device, and, as far as we know, Neuralink does not use it.

CCPM

Another method of measuring the performance of neural interfaces is to measure the number of correctly typed characters per minute (ccpm). This is suitable for both handwriting decoding BCIs and point-and-click BCIs.

According to a paper by Krishna Shenoy’s team, the performance of interfaces that detect handwritten characters is more than 2x higher than that of “point-and-click” interfaces. In addition, such an interface can be reinforced with spell checking, automatic word completion or an intelligent language model, which further increases the typing speed. Below we see a table comparing the performance of various BCIs according to cwpm, ccpm and bitrate, which will be discussed later:

In this table:

• Intra = intracortical – invasive BCI with electrodes in the cerebral cortex (most often some form of Utah Array)
• ReFIT-KF – Recalibrated Feedback Intention-Trained Kalman Filter – specific type of BCI
• HMM – Hidden Markov Model
• CLC – Closed Loop Calibration
• LDA – Linear Discriminant Analysis
• RTI – Retrospective Target Inference
• BS – Brainstem Stroke
• CP – Cerebral Palsy
• DMD – Duchenne Muscular Dystrophy
• SCI – Spinal Cord Injury
• EEG – Electroencephalography
• ECoG – Electrocorticography

Bitrate

For point-and-click BCIs, the best way to measure performance is by using achieved bitrate. This is why Neuralink uses this metric for its N1 Telepathy interface.

Bitrate, measured in bits per second (bps), has no connection to language here and is independent of word completion or prediction algorithms. It is defined as follows:

In the formula:

• B is the achieved bitrate in bits per second (bps)
• N is the number of selectable symbols on the interface (including delete key) and the -1 is because one key is the delete key
• S_c is the number of symbols/characters
• S_i is the number of incorrect symbols/characters
• t is the elapsed measurement time
• the max function prevents the bitrate from being negative, which is not realistic

Here’s the table of results for several BCIs, including the values ​​measured for Neuralink monkey Pager:

Abbreviations in the table:

• NHP means non-human primate
• ReFIT-KF – Recalibrated Feedback Intention-Trained Kalman Filter – specific type of BCI
• HMM – Hidden Markov Model
• Dwell is a delay used instead of decoding clicks

Interfaces generally use one of two methods of selecting targets (clicks). Either “dwell”, i.e. the cursor must stop on a given target for a certain time to be selected, or the interface directly decodes the “click” to select the target, similar to a computer mouse. Click detection generally leads to higher BCI performance than using dwell.

Bitrate can be measured using a grid task such as Neuralink’s WebGrid mentioned above or other random sequences without correlation structure that do not use language, because language has structure (e.g. some letters are more likely to follow the current letter than others).

Neuralink’s record

The highest measured value in Krishna Shenoy’s tables – 6.49 bpm in table no. 2 – was achieved with BCI developed by Professor Paul Nuyujukian. It is the same scientist who in 2021 analyzed a video with macaque Pager on YouTube and was also present at Neuralink’s 2022 Show and tell presentation, where he personally congratulated the company’s specialists on breaking his own record!

So why did Neuralink consider Noland’s interface performance of 4.61 bps (the first image in this article) to be breaking the BCI speed record? It is probably because Neuralink compares its interface solely with point-and-click interfaces, and the 6.49 bpm was achieved with a handwriting decoding BCI.

However, Noland’s current best result of 9.51 bps is an undisputed record among all the aforementioned types of interfaces.

Back to the definition of bit and BCI performance rating

The definition states that “The bit is the most basic unit of information in computing and digital communication. The name is a portmanteau of binary digit. The bit represents a logical state with one of two possible values. These values are most commonly represented as either “1” or “0”, but other representations such as true/false, yes/no, on/off, or +/− are also widely used.”

From the perspective of measuring BCIs performance, the first part of this definition is important – bit is the most basic unit of information. So it seems that with BCIs it is not necessarily about ones and zeros as when measuring data flows in the field of information technology, but rather about a bit more enigmatic “units of information”.

What transmission speeds could Neuralink achieve in the future?

The authors of the study defining the methods for measuring the performance of BCIs mention certain interesting opinion in their text – according to them, the relationship between the number of channels (electrodes) and the interface performance is nonlinear – as the number of channels gets larger, added value of each of them decreases. It would certainly be interesting to know what Neuralink neuroscientists think about this statement.

Elon Musk talks about the need for megabits, perhaps even gigabits per second, for advanced “neural lace” type of BCI. The question is how and when such values ​​could be achieved. Will it be enough to simply increase the number of injected threads and electrodes? Will electrodes need to be injected into all areas of the brain? Will they also have to be placed in deeper parts of the brain?

If the relationship between the number of electrodes and the interface performance is nonlinear, achieving high interface bitrates can be an extremely difficult task.

What is the required data rate for a holistic neural interface?

In the long term, Neuralink wants to create a holistic, high-speed interface for the human brain to fuse human and artificial intelligence. Along the way, it aims to solve different neurological problems and enhance human capabilities.

But what exactly does that mean? How fast does the interface need to be to enable the most advanced brain-computer connection achievable?

DJ Seo, president of Neuralink, offered a possible solution to this question in an interview with Lex Fridman:

There actually is some biological existence proof of what it would take to kind of start to form some of these experiences that may be unique. If you actually look at every one of our brains, there are two hemispheres. There’s a left-sided brain, there’s a right-sided brain. And unless you have some other conditions, you normally don’t feel like left legs or right legs, you just feel like one legs, right? So, what is happening there? Right?

If you actually look at the two hemispheres, there’s a structure that kind of connectorizes the two, called the corpus callosum, that is supposed to have around 200 to 300 million connections or axons. So, does that mean that this is the number of electrodes that we need to have to create some sort of mind meld or a new conscious experience?