NVMe drives in different operating modes of the PCI Express interface: a practical study of interface scalability in data transfer tasks. SSD with PCI Express interface: review and testing of five models What is the PCI Express bus

The PCI Express standard is one of the foundations of modern computers. PCI Express slots have long occupied a strong place on any desktop computer motherboard, displacing other standards, such as PCI. But even the PCI Express standard has its own variations and connection patterns that differ from each other. On new motherboards, starting around 2010, you can see a whole scattering of ports on one motherboard, designated as PCIE or PCI-E, which may differ in the number of lines: one x1 or several x2, x4, x8, x12, x16 and x32.

So, let's find out why there is such confusion among the seemingly simple PCI Express peripheral port. And what is the purpose of each PCI Express x2, x4, x8, x12, x16 and x32 standard?

What is the PCI Express bus?

Back in the 2000s, when the transition took place from the aging PCI standard (extension - interconnection of peripheral components) to PCI Express, the latter had one huge advantage: instead of a serial bus, which was PCI, a point-to-point access bus was used. This meant that each individual PCI port and the cards installed in it could take full advantage of the maximum bandwidth without interfering with each other, as happened with a PCI connection. In those days, the number of peripheral devices inserted into expansion cards was abundant. Network cards, audio cards, TV tuners, and so on - all required a sufficient amount of PC resources. But unlike the PCI standard, which used a common bus for data transfer with multiple devices connected in parallel, PCI Express, when considered in general, is a packet network with a star topology.

PCI Express x16, PCI Express x1 and PCI on one board

In layman's terms, imagine your desktop PC as a small store with one or two salespeople. The old PCI standard was like a grocery store: everyone waited in the same line to be served, experiencing speed issues with the limitation of one salesperson behind the counter. PCI-E is more like a hypermarket: each customer follows his own individual route for groceries, and at the checkout, several cashiers take the order at once.

Obviously, a hypermarket is several times faster than a regular store in terms of speed of service, due to the fact that the store cannot afford the capacity of more than one salesperson with one cash register.

Also with dedicated data lanes for each expansion card or built-in motherboard components.

The influence of the number of lines on throughput

Now, to extend our store and hypermarket metaphor, imagine that each department of the hypermarket has its own cashiers reserved just for them. This is where the idea of ​​multiple data lanes comes into play.

PCI-E has gone through many changes since its inception. These days, new motherboards typically use version 3 of the standard, with the faster version 4 becoming more common, with version 5 expected in 2019. But different versions use the same physical connections, and these connections can be made in four main sizes: x1, x4, x8 and x16. (x32 ports exist, but are extremely rare on regular computer motherboards).

The different physical sizes of PCI-Express ports make it possible to clearly divide them by the number of simultaneous connections to the motherboard: the larger the port is physically, the more maximum connections it can transmit to the card or vice versa. These connections are also called lines. One line can be thought of as a track consisting of two signal pairs: one for sending data and the other for receiving.

Different versions of the PCI-E standard allow different speeds on each lane. But generally speaking, the more lanes there are on a single PCI-E port, the faster data can flow between the peripheral and the rest of the computer.

Returning to our metaphor: if we are talking about one seller in a store, then the x1 strip will be this only seller serving one client. A store with 4 cashiers already has 4 lines x4. And so on, you can assign cashiers by the number of lines, multiplying by 2.

Various PCI Express cards

Types of devices using PCI Express x2, x4, x8, x12, x16 and x32

For the PCI Express 3.0 version, the overall maximum data transfer speed is 8 GT/s. In reality, the speed for the PCI-E 3 version is slightly less than one gigabyte per second per lane.

Thus, a device using the PCI-E x1 port, for example, a low-power sound card or Wi-Fi antenna, will be able to transfer data at a maximum speed of 1 Gbps.

A card that physically fits into a larger slot - x4 or x8, for example, a USB 3.0 expansion card will be able to transfer data four or eight times faster, respectively.

The transfer speed of PCI-E x16 ports is theoretically limited to a maximum bandwidth of about 15 Gbps. This is more than enough in 2017 for all modern graphics cards developed by NVIDIA and AMD.

Most discrete graphics cards use a PCI-E x16 slot

The PCI Express 4.0 protocol allows the use of 16 GT/s, and PCI Express 5.0 will use 32 GT/s.

But currently there are no components that could use this number of lanes with maximum throughput. Modern high-end graphics cards usually use x16 PCI Express 3.0. It makes no sense to use the same lanes for a network card that will only use one lane on the x16 port, since the Ethernet port is only capable of transferring data up to one gigabit per second (which is about one-eighth the throughput of one PCI-E lane - remember: eight bits in one byte).

There are PCI-E SSDs on the market that support the x4 port, but they look set to be replaced by the rapidly evolving new M.2 standard. for SSDs that can also use the PCI-E bus. High-end network cards and enthusiast hardware such as RAID controllers use a combination of x4 and x8 formats.

PCI-E port and lane sizes may vary

This is one of the most confusing problems with PCI-E: a port can be made in the x16 form factor, but not have enough lanes to carry data through, for example, just x4. This is because even though PCI-E can carry an unlimited number of individual connections, there is still a practical limit to the chipset's bandwidth capacity. Cheaper motherboards with lower-end chipsets may only have one x8 slot, even if that slot can physically accommodate an x16 form factor card.

Additionally, motherboards aimed at gamers include up to four full PCI-E slots with x16 and the same number of lanes for maximum bandwidth.

Obviously this can cause problems. If the motherboard has two x16 slots, but one of them only has x4 lanes, then adding a new graphics card will reduce the performance of the first by as much as 75%. This is, of course, only a theoretical result. The architecture of motherboards is such that you will not see a sharp drop in performance.

The correct configuration of two graphics video cards should use exactly two x16 slots if you want maximum comfort from a tandem of two video cards. The manual at the office will help you find out how many lines a particular slot has on your motherboard. manufacturer's website.

Sometimes manufacturers even mark the number of lines on the motherboard PCB next to the slot

You need to know that a shorter x1 or x4 card can physically fit into a longer x8 or x16 slot. The pin configuration of the electrical contacts makes this possible. Naturally, if the card is physically larger than the slot, then you won’t be able to insert it.

Therefore, remember, when purchasing expansion cards or upgrading current ones, you must always remember both the size of the PCI Express slot and the number of lanes required.

If you ask which interface should be used for a solid-state drive that supports the NVMe protocol, then any person (who even knows what NVMe is) will answer: of course PCIe 3.0 x4! True, he will most likely have difficulties with justification. At best, we will get the answer that such drives support PCIe 3.0 x4, and interface bandwidth matters. It is, but all the talk about it began only when some drives in some operations became cramped within the framework of “regular” SATA. But between its 600 MB/s and the (equally theoretical) 4 GB/s of the PCIe 3.0 x4 interface there is simply an abyss, filled with a ton of options! What if one PCIe 3.0 line is enough, since this is already one and a half times larger than SATA600? Adding fuel to the fire are controller manufacturers who are threatening to switch to PCIe 3.0 x2 in budget products, as well as the fact that many users do not have such and such. More precisely, theoretically there are, but they can be released only by reconfiguring the system or even changing something in it that you don’t want to do. But I want to buy a top-end solid-state drive, but there are fears that there will be no benefit at all from this (even moral satisfaction from the results of test utilities).

But is this true or not? In other words, is it really necessary to focus exclusively on the supported operating mode - or is it still possible in practice? give up principles? This is exactly what we decided to check today. Let the check be quick and not pretend to be exhaustive, but the information received should be enough (as it seems to us) at least to think about it... For now, let's briefly get acquainted with the theory.

PCI Express: existing standards and their bandwidth

Let's start with what PCIe is and at what speed this interface operates. It is often called a “bus,” which is somewhat ideologically incorrect: as such, there is no bus to which all devices are connected. In reality there is a set of point-to-point connections (similar to many other serial interfaces) with a controller in the middle and devices attached to it (each of which could itself be a next-level hub).

The first version of PCI Express appeared almost 15 years ago. The focus on use inside a computer (often within the same board) made it possible to make the standard high-speed: 2.5 gigatransactions per second. Because the interface is serial and full-duplex, a single PCIe lane (x1; effectively an atomic unit) provides data transfer speeds of up to 5 Gbps. However, in each direction it is only half of this, i.e. 2.5 Gbps, and this is the full speed of the interface, not the “useful” one: to improve reliability, each byte is encoded with 10 bits, so the theoretical throughput of one PCIe lane 1.x is approximately 250 MB/s each way. In practice, it is still necessary to transfer service information, and in the end it is more correct to talk about ≈200 MB/s of user data transfer. Which, however, at that time not only covered the needs of most devices, but also provided a solid reserve: just remember that the predecessor of PCIe in the segment of mass system interfaces, namely the PCI bus, provided a throughput of 133 MB/s. And even if we consider not only mass implementation, but also all PCI options, the maximum was 533 MB/s, and for the entire bus, i.e., such a PS was divided into all devices connected to it. Here, 250 MB/s (since for PCI, too, the total and not the useful throughput is usually given) per line - in exclusive use. And for devices that need more, it was initially possible to aggregate several lines into a single interface, in powers of two - from 2 to 32, i.e., the x32 version provided for by the standard could transmit up to 8 GB/s in each direction. In personal computers, x32 was not used due to the complexity of creating and wiring the corresponding controllers and devices, so the maximum option was 16 lines. It was (and is still used) mainly by video cards, since most devices do not require so much. In general, for a considerable number of them, one line is enough, but some successfully use both x4 and x8: just on the storage topic - RAID controllers or SSDs.

Time did not stand still, and about 10 years ago the second version of PCIe appeared. The improvements were not only about speeds, but a step forward was also taken in this regard - the interface began to provide 5 gigatransactions per second while maintaining the same encoding scheme, i.e., the throughput was doubled. And it doubled again in 2010: PCIe 3.0 provides 8 (rather than 10) gigatransactions per second, but the redundancy has been reduced - now 130 bits are used to encode 128, not 160 as before. In principle, the PCIe 4.0 version with another doubling of speeds is already ready to appear on paper, but we are unlikely to see it in hardware in the near future. In fact, PCIe 3.0 is still used in many platforms in conjunction with PCIe 2.0, because the performance of the latter is simply... not needed for many applications. And where needed, the good old method of line aggregation works. Only each of them has become four times faster over the past years, i.e. PCIe 3.0 x4 is PCIe 1.0 x16, the fastest slot in computers of the mid-2000s. This option is supported by top-end SSD controllers, and it is recommended to use it. It is clear that if such an opportunity exists, a lot is not a little. What if she doesn't exist? Will there be any problems, and if so, what are they? This is the question we have to deal with.

Testing methodology

It is not difficult to carry out tests with different versions of the PCIe standard: almost all controllers allow you to use not only the one they support, but also all earlier ones. It’s more difficult with the number of lanes: we wanted to directly test options with one or two PCIe lanes. The Asus H97-Pro Gamer board on the Intel H97 chipset that we usually use does not support the full set, but in addition to the x16 “processor” slot (which is usually used), it has another one that operates in PCIe 2.0 x2 or x4 modes. We used this trio, adding to it the PCIe 2.0 “processor” slot mode in order to evaluate whether there was a difference. Still, in this case, there are no extraneous “intermediaries” between the processor and the SSD, but when working with a “chipset” slot, there is: the chipset itself, which is actually connected to the processor by the same PCIe 2.0 x4. It was possible to add several more operating modes, but we were still going to conduct the main part of the study on another system.

The fact is that we decided to take this opportunity and at the same time check one “urban legend”, namely the belief about the usefulness of using top processors for testing drives. So we took the eight-core Core i7-5960X - a relative of the Core i3-4170 usually used in tests (these are Haswell and Haswell-E), but which has four times more cores. In addition, the Asus Sabertooth X99 board found in the bins is useful to us today due to the presence of a PCIe x4 slot, which in fact can work as x1 or x2. In this system, we tested three x4 options (PCIe 1.0/2.0/3.0) from the processor and chipset PCIe 1.0 x1, PCIe 1.0 x2, PCIe 2.0 x1 and PCIe 2.0 x2 (in all cases, chipset configurations are marked in the diagrams with (c)). Does it make sense to turn to the first version of PCIe now, given the fact that there is hardly a single board that supports only this version of the standard and can boot from an NVMe device? From a practical point of view, no, but to check the a priori assumed ratio of PCIe 1.1 x4 = PCIe 2.0 x2 and the like, it will be useful to us. If the test shows that the bus scalability corresponds to the theory, then it doesn’t matter that we have not yet been able to obtain practically significant ways to connect PCIe 3.0 x1/x2: the first will be identical to PCIe 1.1 x4 or PCIe 2.0 x2, and the second - PCIe 2.0 x4 . And we have them.

In terms of software, we limited ourselves to only Anvil’s Storage Utilities 1.1.0: it measures a variety of low-level characteristics of drives quite well, and we don’t need anything else. Quite the contrary: any influence of other components of the system is extremely undesirable, so low-level synthetics for our purposes have no alternative.

We used a 240 GB Patriot Hellfire as a “working fluid”. As it was established during testing, this is not a performance record-holder, but its speed characteristics are quite consistent with the results of the best SSDs of the same class and the same capacity. Yes, and there are already slower devices on the market, and there will be more and more of them. In principle, it would be possible to repeat the tests with something faster, but, in our opinion, there is no need for this - the results are predictable. But let’s not get ahead of ourselves, but let’s see what we got.

Test results

When testing Hellfire, we noticed that the maximum speed for sequential operations can only be “squeezed out” of a multi-threaded load, so this also needs to be taken into account for the future: the theoretical throughput is only theoretical, because the “real” data received in different programs under different scenarios will no longer depend on it, but on these very programs and scenarios - in the case, of course, when force majeure circumstances do not interfere :) These are exactly the circumstances we are now observing: it has already been said above that PCIe 1.x x1 is ≈200 MB/s, and that's exactly what we see. Two PCIe 1.x lanes or one PCIe 2.0 lanes are twice as fast, and that's exactly what we're seeing. Four PCIe 1.x lanes, two PCIe 2.0 or one PCIe 3.0 are even twice as fast, which was confirmed for the first two options, so the third is unlikely to be different. That is, in principle, scalability, as expected, is ideal: the operations are linear, flash handles them well, so the interface matters. Flash stops cope well to PCIe 2.0 x4 for recording (which means PCIe 3.0 x2 is also suitable). Reading “may” be more, but the last step already gives one and a half, and not twofold (as it potentially should be) increase. We also note that there is no noticeable difference between the chipset and processor controllers, and between platforms as well. However, LGA2011-3 is a little ahead, but only slightly.

Everything is smooth and beautiful. But does not tear templates: the maximum in these tests is only slightly more than 500 MB/s, and this is quite capable even of SATA600 or (in the application to today's testing) PCIe 1.0 x4 / PCIe 2.0 x2 / PCIe 3.0 x1. That’s right: don’t be alarmed by the release of budget controllers for PCIe x2 or the presence of only so many lines (and the 2.0 version of the standard) in the M.2 slots on some boards when more is not needed. Sometimes you don’t need that much: the maximum results were achieved with a queue of 16 commands, which is not typical for mass-produced software. More often there is a queue with 1-4 commands, and for this you can get by with one line of the very first PCIe and even the very first SATA. However, there are overheads and other things, so a fast interface is useful. However, being too fast is perhaps not harmful.

Also, in this test the platforms behave differently, and with a single command queue - fundamentally differently. The “trouble” is not that many cores are bad. They are not used here anyway, except perhaps one, and not so much that the boost mode is fully deployed. So we have a difference of about 20% in core frequency and one and a half times in cache memory - in Haswell-E it operates at a lower frequency, and not synchronously with the cores. In general, a top-end platform can only be useful for knocking out the maximum “Yops” through the most multi-threaded mode with a large command queue depth. The only pity is that from the point of view of practical work, this is completely spherical synthetics in a vacuum :)

On the recording, the situation has not changed fundamentally - in every sense. But what’s funny is that on both systems the PCIe 2.0 x4 mode in the “processor” slot turned out to be the fastest. On both! And with multiple checks/rechecks. At this point you can’t help but think about whether you need these are your new standards Or is it better not to rush anywhere at all...

When working with blocks of different sizes, the theoretical idyll is shattered by the fact that increasing the speed of the interface still makes sense. The resulting figures are such that a couple of PCIe 2.0 lanes would be enough, but in reality in this case the performance is lower than that of PCIe 3.0 x4, albeit not by several times. And in general, here the budget platform “clogs” the top one to a much greater extent. But it is precisely this kind of operation that is found mainly in application software, i.e. this diagram is the closest to reality. As a result, it is not surprising that thick interfaces and fashionable protocols do not provide any “wow” effect. More precisely, those switching from mechanics will be given, but exactly the same as any solid-state drive with any interface will provide him.

Total

To make it easier to perceive the picture of the hospital as a whole, we used the score given by the program (total - for reading and writing), normalizing it according to the PCIe 2.0 x4 “chipset” mode: at the moment it is the most widely available, since it is found even on LGA1155 or AMD platforms without the need to “offend” the video card. In addition, it is equivalent to PCIe 3.0 x2, which budget controllers are preparing to master. And on the new AMD AM4 platform, again, this is exactly the mode that can be obtained without affecting the discrete video card.

So what do we see? The use of PCIe 3.0 x4, if possible, is certainly preferable, but not necessary: ​​it brings literally 10% additional performance to mid-class NVMe drives (in its initially top segment). And even then - due to operations that, in general, are not so often encountered in practice. Why is this particular option implemented in this case? Firstly, there was such an opportunity, but the reserve is not enough for the pocket. Secondly, there are drives even faster than our test Patriot Hellfire. Thirdly, there are areas of activity where loads that are “atypical” for a desktop system are quite typical. Moreover, this is where the performance of the data storage system, or at least the ability to make part of it very fast, is most critical. But this does not apply to ordinary personal computers.

In them, as we see, the use of PCIe 2.0 x2 (or, accordingly, PCIe 3.0 x1) does not lead to a dramatic decrease in performance - only by 15-20%. And this despite the fact that in this case we limited the potential capabilities of the controller by four times! For many operations this throughput is sufficient. One PCIe 2.0 line is no longer enough, so it makes sense for controllers to support PCIe 3.0 - and given the severe shortage of lines in a modern system, this will work well. In addition, x4 width is useful - even if there is no support for modern versions of PCIe in the system, it will still allow you to work at normal speed (albeit slower than it could potentially) if there is a more or less wide slot.

In principle, a large number of scenarios in which the flash memory itself turns out to be the bottleneck (yes, this is possible and is inherent not only in mechanics) leads to the fact that the four lanes of the third version of PCIe on this drive are about 3.5 times faster than the first one - the theoretical throughput of these two cases differs by 16 times. Which, of course, does not mean that you need to rush to master very slow interfaces - their time is gone forever. It’s just that many of the possibilities of fast interfaces can only be realized in the future. Or in conditions that an ordinary user of an ordinary computer will never directly encounter in his life (with the exception of those who like to compare themselves with who knows what). Actually, that's all.

PCI Express, whose full technical name is "Peripheral Component Interconnect Express" but often abbreviated as PCIe or PCI-E, is a standard connection type for internal devices such as video cards, sound cards, wifi adapters and other peripheral devices on a personal computer.

Let's understand the differences between the PCI-E connector.

Typically, this high-speed port refers to the actual expansion slots on the motherboard that accept traditional PCIe-based expansion cards and expansion card types.

PCI Express has virtually replaced PCI, both of which have replaced the oldest widely used connection type, called ISA. Although PCs can contain a variety of expansion slots, PCI Express is considered the standard internal interface for the fastest slot. Today, many personal computer motherboards are manufactured only with PCI Express connectors.

How does PCI Express work?

Like older standards like PCI and AGP, the Express-based device physically goes into a high-speed connector on the motherboard.

The interface of this connector provides high-speed communication between the device and, as well as other equipment.

Although not very common, there is also an external version of the high-speed port, unsurprisingly called External PCI Express, but often shortened to PCIe. ePCIe devices that are external require a special cable to connect any external PCIe device to the PC through a PCIe port, usually located on the back of the PC, supplied either by the motherboard or a dedicated internal PCIe card.

What types of PCI Express cards are there?

With the demand for faster, more realistic video games and video editing tools, graphics cards were the first types of computer peripherals to take advantage of the benefits offered directly by PCIe.

While graphics cards are still the most common type of PCIe card, you'll find other devices that connect to the motherboard, processor, and RAM much faster. It is also increasingly common to make PCIe connections instead of conventional PCI. For example, many high-end sound cards now use a high-speed port, as well as an increasing number of wired and wireless network interface cards.

Of course, with PCIe being replaced by PCI and AGP entirely on newer motherboards, almost every type of internal expansion card based on older interfaces is being rebuilt to be able to use the PCI Express bus. This includes things like expansion cards, Bluetooth cards, etc.

What are the different PCI Express formats?

Express x1 ... Express 3.0 ... Express x16. What does "x" mean? How do you know if your PC supports it? If there is a PCI Express x1 card, and there is only an Express x16 slot, does it work compatible? If not, what are your options?

It's often not entirely clear when you buy an expansion card for your computer, such as a new graphics card, which of the various PCIe technologies works better with your PC than the other. However, as complex as it is, it looks quite simple once you understand two important pieces of information about a high-speed port: the part describing the physical size and the part describing the technology version, as described below.

PCIe sizes: x16, x8, x4, and x1

As the title suggests, the number after the x indicates the physical size of the PCI-E card or slot, with x16 being the largest and x1 the smallest.

Here's how the different sizes are formed:

Regardless of the size of the high-speed port or card, the key notch, that small space in the card or slot, is always located at pin 11. That is, the length of pin 11 continues to increase as we move from PCIe x1 to PCIe x16. This allows you to flexibly use cards of one size with slots of another.

PCIe cards fit in any high-performance port slot on the motherboard that is at least as large. For example, a PCIe x1 card will fit into any PCIe x4, PCIe x8, or PCIe x16 slot. The PCIe x8 card will fit into any PCIe x8 or PCIe x16 slot. PCIe cards that are larger than a PCIe slot can fit into the smaller slot, but only if that PCI-E slot is open (i.e. does not have a plug at the end of the slot).

In general, a larger Express card or slot will support more performance, assuming the two cards or slots you're comparing support the same version of PCIe.

PCIe version: 4.0, 3.0, 2.0 and 1.0

Any number after PCIe that you find on a device or motherboard indicates the latest version number of the PCI Express specification being used.

Here's how the different versions of the PCI Express controller compare:

All versions of the high-speed port are backwards and forwards compatible, meaning no matter what version your PCIe card or motherboard supports, they should work together at least at a minimum level. As you can see, major updates to the port standard dramatically increase throughput each time, greatly increasing the potential of what the associated hardware can do.

The version's improvements also include bug fixes, added features, and improved power management, but the increase in bandwidth is the most important change to note from version to version.

Maximizing compatibility with PCIe

As you read in the sizes and versions sections above, uses almost any configuration you can imagine. If it's physically fit, it probably works...that's great. However, it's important to know that to increase bandwidth (which usually equates to maximum performance), you need to select the highest PCIe version supported by your motherboard and select the largest size of a given port that will fit.

For example, a graphics card with a 3.0 x16 high-speed port will give you maximum performance, but only if the motherboard supports the 3.0 high-speed port and has a free x16 high-speed port. If the motherboard model uses PCIe 2.0 exclusively, the card will only run at the supported speed (for example, 64 Gbps in an x16 slot).

Most motherboards and personal computers released in 2013 or later likely support Express v3.0. If you are unsure, check your motherboard or PC manual. If you can't find any definitive information about the PCI version your motherboard can use, I recommend buying the largest and latest version of PCIe card, if it fits, of course.

What will replace PCIe?

Video game developers are always looking for games that become increasingly more realistic, but can only do this if they can transfer more data from their game programs to a VR headset or PC screen, and this requires faster interfaces. Because of this, PCI Express will not continue to dominate its laurels. PCI Express 3.0 is amazingly fast, but the world is striving for incredibly fast transfers.

PCI Express 5.0, due to be completed by 2019, will use 31,504 gigabits per second of bandwidth per lane (3,938 megabytes per second), twice as much as the high-speed slot version 4.0 offers. There are a number of other interface standards other than PCIe that the tech industry is looking at, but since they will require major hardware changes, PCIe looks set to remain the leader for some, a long time as the fastest ever.