PCI-e bifurcation explained

OK, some asked about ‘what is bifurcation’ from the previous post. Essentially, if you have a PCI-e x8 slot, you can split it in half and make it 2 x4 slots. If you have a x16, you can make it 1 x8 and 2×4, or 4×4.

You can see below, i’ve overlayed my BIOS setup on top of the motherboard diagram (here a SuperMicro x10DRi-LN4+). Now, if your BIOS doesn’t have a bifurc option, you can possibly get it to do so by adding support into the BIOS. I’m not going to help you with this, its very complex, but I was able to add both UEFI NVME boot and bifurcation to a different SuperMicro motherboard by adding the UEFI modules into it manually. YMMV. Void where prohibited.

So in my case, I have a NVME carrier which is capable of holding 4 NVME drives. It is passive (no PCI bridge is onboard). This means that it is conceptually 4 PCI-E x4 drives. Without bifurcation, it just won’t work. Some people refer to this as ‘pci splitting’. You may see references to this in the ‘crypto-mining’ industry, where people are using 1x interfaces via cables to mining ASIC.

Be careful here, bifurcation is supported on server motherboards with modern chipsets, but its support on desktops is not as universal. And just because your motherboard supports it doesn’t mean your BIOS will.

It may also have downstream affects on other PCI-e cards, e.g. reducing their lane-width. Caveat Emptor.

EDIT (2021-01-24)

There have been a lot of questions on this article since I published it. I’ll cover a few off here.

  1. If your add-in card has a bridge on it (a chip with many pins), you do not need PCI Bifurcation.
  2. If your add-in card has no-bridge, but exposes >1 device (e.g. a 2 NVME carrier), its just a fancy way of wiring. Take a /8 slot, make 2 /4 slots. It cannot make 4 /4 slots out of a single /8. To do that you need a bridge.
  3. If your BIOS does not expose bifurcation, you will probably find that 1 of the devices on your Add-In carrd works (e.g. NVME0) but not the others
  4. If you want to boot off of NVME, your BIOS needs to support it. In practice, this means UEFI support. Most new ones do, older ones can sometimes be modified to add UEFI NVME support
  5. Bifurcation does not change performance. Its not faster, its not slower. Its really the same concept as just wiring more smaller slots.
  6. The Bridge vs No-Bridge on an Add-In card… a bridge uses more power, adds latency, reduces performance, but is more compatible.
  7. Does my chipset support bifurcation? Intel’s Ark is a great resource. Just search the part number and you will get simple data to read.
  8. Is bifurcation for me? I would say the main driver is adding more NVME drives to your machine. As their size has increased and their cost has decreased, this has become somewhat less interesting I guess. The other use case, crypto-mining, has lots of material online.

Posted

in

by

Tags:

Comments

49 Responses to “PCI-e bifurcation explained”

  1. Thanks for such easy and simple explanation. This means a PCIe x16 slot can be run in 8x mode but what if we have two PCIe x16 slots?

    1. db

      usually you can take a /x16 into 4 x /x4, or 2 x /x8, etc. If you have 2 pcie x16, you can convert each of them.
      It may be tricky to understand which one runs to which physical slot, so experimentation / motherboard manual may be required.

      Not all chipset/motherboard bios expose bifurcation, but if its there, its helpful.

      1. Tor Bruheim

        Question: If I have 4 nvme drives, the bandwidth with 4×4 bifurcation means I have only a single PCIe4 bandwidth (3.94GB/s) per nvme drive? It makes seance for me seeing it that way.

        1. db

          bifurcation does not reduce bandwidth.
          if you have a /x16, and 4 x4 drives, each gets x4 of the bandwidth.

  2. Milan

    Hello DB,

    Very nice article. To me the most interesting part is the one you havent explaind…how to make BIOS allow bifurcation even if manufacturer didn’t want you to do it. If you can spare some time I am in a greatly need of your help.

    This is the situation. I have a Supermicro X11SSZ-F mobo, AOC-SLG3-2M2 card and two Corsair MP510 NVMes. 2M2 card is in the Slot6 x16.

    As you can imagine, mobo doesn’t see two NVMes, just one, and with newest version of bios there is still no option to bifurcate x16 slot to x4x4x4x4 (or to any other value).

    By now I read more that 20 posts about this issue, and there are lots of solutions but none can apply to X11SSZ-F…

    Can you please help me to make both of NVMes work…?

    Thank you in advance!

    1. db

      So I’d rather not give very specific instructions since some might brick 🙂

      https://www.win-raid.com/t871f50-Guide-How-to-get-full-NVMe-support-for-all-Systems-with-an-AMI-UEFI-BIOS.html gives u a general starting point.
      https://forums.servethehome.com/index.php?threads/nvme-boot-with-supermicro-x9da7-x9dri-f.13245/

      I had no problems. I used AMI Bios 2.0b 2017-08-19 on X10DRi-LN4+, and asus hyper m.2 x16 card as 4 x NVME /x4. And all worked well.

      I also did this on AMI BIOS 4801 2014-07-25 on Asus p9x79 pro.

      in a nutshell, the tools linked allow you to unpack the bios. You then add a uefi module and repack. The IPMI allows programing the bios. if you ‘brick’ you can still program it but might need a key from supermicro.

      –don

      1. Hello!
        I really liked your article and the explanation in it!
        I have the same problem only P9X79 Delux motherboard. Could you help with the BIOS for it with bifurcation support on PCIe for installing the DELL UltraSpeed ​​4xSSD board, it is the same as asus hyper m.2.
        I changed the BIOS at https://www.win-raid.com/t2790f44-OFFER-ASUS-P-X-DELUXE-BIOS-MOD-NVME-Dxe.html#msg38148.
        But only one of the 4 installed drives is visible.
        I would be very grateful for your help if you can give the BIOS firmware ready for work.
        Thank.

      2. Allen Thompson

        Don,
        I see your post is a few years old but have to ask as I am not good with code. I am purchasing the X10DRi-LN4+ MB in a 2U server on Ebay for a good price but would really like to add some NVME’s as bootable drives. Would you happen to have the BIOS update lying around that you could forward to me? I know the risk and will not hold ill will if it does not work or bricks my system. Thanks in advance for any advice. Allen T.

        1. db

          sorry, i added the boot support to a different supermicro, not my X10DRi-LN4+.
          On my X10DRi-LN4+ i use bifurcation but boot from a usb-header flash.
          You just need ‘grub’ or systemd-boot on that device, from there it can go to the nvme.

          https://www.amazon.de/-/en/dp/B076SH5KCG

          is the device i used w/ the X10… mobo.

          i think it will be possible to edit the bios and add the uefi nvme module, but don’t have time to try it, sorry.
          the ones that worked for me were x9 something gen and also a (non supermicro) desktop motherboard.

          –don

  3. Aleksey

    Hello!
    I really liked your article and the explanation in it!
    I have the same problem only P9X79 Delux motherboard. Could you help with the BIOS for it with bifurcation support on PCIe for installing the DELL UltraSpeed ​​4xSSD board, it is the same as asus hyper m.2.
    I changed the BIOS at https://www.win-raid.com/t2790f44-OFFER-ASUS-P-X-DELUXE-BIOS-MOD-NVME-Dxe.html#msg38148.
    But only one of the 4 installed drives is visible.
    I would be very grateful for your help if you can give the BIOS firmware ready for work.
    Thank.

    1. db

      https://www.win-raid.com/t4252f16-GUIDE-Adding-Bifurcation-Support-to-ASUS-X-UEFI-BIOS.html
      seems to talk about this (the link you found)
      My desktop is also p9x79-based, but i’m retiring it as soon as i can get the new ryzen 3950x, and it doesn’t run nvme, so i can’t really test this.
      I’m assuming the DELL UltraSpeed ​​4xSSD is non-bridged based.
      if you see only one of the installed drives, does it matter which /x16 slot you place it in? E.g. does it work differently in another slot?

  4. Timothy Ramich

    I am wondering. I have the AOC-SHG3-4M2P card. It is an x8 card. If bifurcation is for splitting up a slot, then why does the manual for this card mention setting the bifurcation for that slot to x4x4x4x4? That’s x16… It has a switch on it to combine what is literally x16 between the four SSDs down to x8. What am I not understanding?

    1. db

      bifurcation splits a bridge.
      on most systems, the only bridge is in the processor (formerly the north/south bridge).
      If you have a card w/ a bridge (like a PLX) on it, that in turn can be bifurcated if the bios or other setup supports.

      you might have a /x8 upstream and a /16 downstream. E.g. it might present /8 to the processor side, and 4×4 to the nvme side. This would be a good thing, e.g. it would be oversubscribed (so not all 4 cards can operate full speed simultaneously), but, in burst, all of them can sometimes operate full speed without using a /16 upstream.

  5. Triple JJJ

    Question, I have an SuperMicro X10drg-q, it seems it does not have bifurcation, what if I use this Asus M.2 Hyper thingy, and I had 2, will it work ok if I did it through software mode(Within Windows/Stripped)?

    I don’t think that’s the issue, what if I use all 4 Nvme’s WITHOUT bifurcation? Can I still raid 0 them through Windows????

    Thanks again.

    1. db

      if you have a M.2 card with no PCI bridge (e.g. it doesn’t have a big chip on it), and it has >1 NVME slot, then you must enable bifurcation.

      bifurcation means splitting the big slot into little slots.

      There are single NVME expander cards, e.g. 1 NVME, they don’t need this.
      As for RAID-0 stripe, as long as you can see the cards, you can do this.

      as for your motherboard, its materially similar to the one i posted about, same chipset. Maybe you just don’t have the bios updated?
      https://forums.unraid.net/topic/87033-asus-hyper-x16-nvme-card-x10drg-q/
      suggests that it will work.

  6. Raymond McLaughlin

    My Understanding at the moment?
    Bifurcation is the ability to assign PCIe lane functions. It can take a 16x lane slot and divide the lanes into 2 – 8x lanes or 4 – 4x lanes.  My Asus x99 motherboard has this ability to suport bifunction and my gigabyte x99 udp 3 motherboard  does not. The expansion card in question ( NV95NF ) does not support bifunction but requires the 16x slot to function at 4x x 4. 
    The Gigabyte bios only has a auto, gen 1, gen 2, or gen 3 setting. I can not tell if the auto would recognize the lane usage automatically for the NV95NF (https://store.sfpcables.com/nvme-ssd-adapter-card-pcie-x16-to-4x-m-2-nvme-connectors) card to run all four m.2 modules. 
    Opinions welcome and facts are GOLD! Thanks!

    1. db

      so indeed the card you linked here requires a chipset + bios supporting bifurcation: there is no bridge on it.
      So the processor is putting a /16 bus out to that slot, you need to convince it to make that 4 x /4 slots.

      this link suggests a way you can do it for what might be the same motherboard you have.
      https://www.techpowerup.com/forums/threads/how-on-the-x99-platform-i-got-16x-16x-4x-4x-m-2-pci-e-operation.240074/

      it is similar to the modification to the bios i had to make on asus motherboard.

      good luck.

      if you cannot get the bifurcation working, 1 of those 4 nvme slots on the expansion card will still work, not sure which one.

  7. Mike

    I have a question regarding pcie4 bifurcation and x8 pcie2/3 cards (like video capture ones that require x8 connection)

    If I’ll have pcie4 x4 on the x16 slot(assigned by chipset, x570) does it mean only x4pins work or is there hijink on the CPU/chipset side that makes is effectively pcie3x8 with all pins used correctly(x8 pcie2/3 is what I need)?

    1. db

      if you have a /16 slot, you can split it into 2 x /8. Or 4 x /4.
      I suppose it might be possible to do 1 /8 and 2/4, but i’m not sure.

      Its about addressing and bandwidth. Bifurcation is a split: it neither creates nor destroys bandwidth, each lane(pin) still has the same intrinsic bandwidth.

      out of your processor comes N lanes. The mother board manufacturer could have made N /1 slots, or N/2 /2 slots, or N/4 /4 slots, etc. Or any combination.

      What bifurcation does is allow you to override (downards) the decision they made. It is instructing the bios that, the motherboard manufacturer is wrong, its not a single /16 slot, its e.g. 4 /4. This is now just like they had wired 4 connectors.

      if you need /8 pcie, then you can split your /16 in half and use that.

      Note: you do not need to enable bifurcation unless you have a card w/ multiple devics and no bridge.

      If you have a single device on a /8 card, feel free to plug it into the /16 slot with no additional configuration. it will still be /8 speed, and the other lines are unused. But this is part of the standard.

      Bifurcation only really makes sense w/ e.g. NVME carrier cards. Perhaps some network cards.

  8. Jasko Dedic

    Thank you for very simple and right on the point explanation
    Can you please clarify can I then use x16 on my Mac Pro 5.1 for ASUS HYPER M.2 PCIe x16 NVMe VROC RAID Card V2. Now I have only one of for NVMe 1TB working and is plugged in x4
    You are first who explained problem, and I was trying to find info for long time
    Thank you in advance and best regards

    1. db

      So i’m not familiar with the mac pro bios-equivalent.
      From a hardware technology standpoint, yes, you can use the asus hyper m.2 (these are what i have), with 4 nvme from a host /x16 slot.

      As to whether the mac enables the bifurcation or not, that is another story.
      https://www.youtube.com/watch?v=UYaOfUyuD3k
      suggests no. If you want to be safe, get a carrier card that has a bridge. You will lose a small amount of performance, but it will always work.
      (e.g. these https://www.amfeltec.com/pci-express-carrier-boards-for-m-2-ssd-modules/).
      A bridge will be a largish chip somewhere on the PCB, perhaps w/ a heatsink.

      1. Jasko Dedic

        Thank you very much For your help
        Best regards

  9. a5cent

    “Be careful here, bifurcation is supported on server motherboards with modern chipsets”

    As you stated and as indicated by AMD’s own materials, CPU PCIe bifurcation is dependent on the chipset:

    https://i0.wp.com/www.techarp.com/wp-content/uploads/2018/07/AMD-B450-chipset-slide-11.jpg?ssl=1

    But why? The PCIe controller is in the CPU itself. The PCIe bus off the CPU leads directly to the PCIe slot. The chipset isn’t involved at all. However, if I insert the exact same Ryzen 3000 CPU into a motherboard with either a B450 or an X570 chipset, PCIe bifurcation off the CPU will only work on the later motherboard.

    Is this a market segmentation tactic, where AMD disables a feature when paired with a lower end chipset, although theoretically it would work perfectly fine on B450 too?

    1. db

      Good question. Dunno.
      Perhaps relates to interrupt routing? clock routing?

    2. Enum

      Not sure, but if BIOS needs to support specific PCIe device enumeration for that feature, it opens doors to market segmentation.

  10. Liang

    This board X10DRi-LN4+ supports 2 PCI-E 3.0 x16, 3 PCI-E 3.0 x8, and 1 PCI-E 2.0 x4 (in x8) slot, I’m thinking if it’s possible to make it 7 /x8 for 7 graphic cards with some pice riser cards / extension cables.

  11. Enum

    Interesting article. There are some official references the storage devices are supported on some boards which are not even a server motherboards:
    https://www.asus.com/support/FAQ/1037507

    I am curious if the bifurcation could work with GPU instead of just storage devices.

    1. db

      pci bifurcation works for any device type. The BIOS doesn’t know about the card you plugin. Its just splitting the lanes and treating the endpoints.

      Indeed many, but not all, non-server motherboards support. Get yours today!

  12. Peter Hindes

    I was wondering if it would be possible to split a 16x lane into 16* 1x lanes? I am interested in an overkill setup with a bunch of m.2 drives that don’t individually need too much bandwidth. I have not seen anyone talk about this possibility. I would assume that no bios or motherboard support exists for that type of configuration, but maybe there is an active chip that can handle it. I have heard of some active bifurcation adaptors. What do you think? Is 16* 1x even supported in the pcie spec?

    1. db

      you sometimes see that w/ bitcoin miners, although more commonly the reverse, (/x1 to /x16 to trick a graphics card).

      The short answer is… yes, it is possible, but i doubt you will find a bios that agrees with the concept.

  13. Diego

    Hi, Can a pcie 4.0 x8 be splitted into 2x pcie3.0x8??

    1. db

      the splitting decreases the lanes.
      So a x8 can be split into 2 x4. But not 2 x8. What you are suggesting is achieved with a PCI bridge, an active silicon element, rather than just wiring.

      PCI bifurcation is nothing more than rewiring the wires and teaching the host bridge what you’ve done from a slot# perspective.

      as for PCI4 to PCI3, bifurcation is a wiring technique, and PCI4 is backwards compatible, so splitting a PCI4 x8 to 2 x PCI4 x4 will work, and a PCI3 card in one of them will work too.

      The short answer is always: does my BIOS expose the option.For most people, if you don’t see a BIOS option to do this, you cannot (since it would mean patching the BIOS which is out of scope).

      If you want to take a slot and split it, and don’t have the bios option, then the card you buy should have a bridge on it. E.g. if you want to put 2 NVME in that slot, you can buy a passive card if you have bios support for bifurcation, or, you can buy a card with a bridge to be safe.

      you will get higher performance and lower power if you can avoid the bridge (cheaper too).

      I’m blurring “motherboard supports” and “bios supports” here since to most people the are the same outcome.

  14. Pramod Nirwan

    Hello There,

    Good day!

    I am using the X10DAi motherboard with Bios 1.0b, and Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz 2.60 GHz (2 processors).
    I’m planning to buy ASUS Hyper M.2 X16 PCIe 3.0 X4 Expansion Card V2 Supports 4 NVMe M.2 (https://www.amazon.ae/gp/product/B07NQBQB6Z/ref=ox_sc_act_title_2?smid=AUDLGVMX87PKQ&psc=1)
    I want to install 4 Nvme M.2. because I want to make an 8TB drive.
    I have 2 Questions.
    1. Is my motherboard / CPU supports Intel Virtual RAID on CPU (VROC)? Is it necessary?
    2. What will be the PCI-E bifurcation process in order to use 4 m.2 Nvme.? I have PCI-E 3 * 16 slot free.

    In BIOS I can only change Burification 4*4 and not 4*4*4*4.

    Please help me out with it. I did a lot of research but can’t find the solution.

    Thanks in advance!
    Pramod

    1. db

      i don’t know what intel vroc is, can’t help you on that.

      your motherboard and bios should support bifurcation. for a /16 slot, it should support 4x4x4x4, and for a /8 slot it should support 4×4.

      The screenshot i gave shows mine, which is very similar.
      so if you put the asus card in, split the slot into 4, it should just work.

      1. Pramod Nirwan

        Thanks for your reply!
        Problem solved. After I update the BIOS, now my motherboard support bifurcation for a /16 slot, before it was showing only 4*4

  15. Jayme Snyder

    I have tried to enable bifurcation on my desktop which is an Intel W2600CR2. Sure it’s old, but it’s got 24 cores and works well for it’s age.
    I’d just like to add a second NVMe drive to my Asus Hyper M.2 Gen 4 carrier card…

    In the AMI BCP I don’t find any string or menu to hint at bifurcation support..
    The single efi firmware file actually supports a number of different Intel motherboards. The cap file contains dxe driver images named like “${IntelCodeName}PlatformInfo” and so there is one for my board: “CrownPassPlatformInfo” – I suspect this is where the IIO configuration registers are set for the relevant motherboard.

    Wouldn’t the firmware option just be setting PCIE_IOU_BIF_CTRL in the Processor Integrated IO configuration registers based on the physical limitations imposed by the board traces?
    Couldn’t a dxe driver be loaded at the right time that sets the configuration registers?

    1. db

      https://ark.intel.com/content/www/us/en/ark/products/56338/intel-workstation-board-w2600cr2.html

      C602 chipset.

      So i’m running a supermicro w/ C602 chipset and it supports bifurcation.
      feeling lucky?
      https://www.win-raid.com/t4252f16-GUIDE-Adding-Bifurcation-Support-to-ASUS-X-UEFI-BIOS.html

      you’re welcome.

      It worked for me on my asus desktop board.

      1. Jayme Snyder

        Yeah I had already read a bit into that thread. I know the chipset supports bifurcation…
        My BIOS’s configuration parameters do not look like they do for the Asus AMI BIOS. My same BIOS file is used on this 2p workstation board, micro atx board, 1u server board… etc.
        It seems like most OEMs (MSI/ASUS/Supermicro) build a board, then branch, customize and released a board specific BIOS hiding irrelevant parameters to their generic PCI conf module…
        Not so in this Intel BIOS file… the optimal/failsafe BCP has to support radically different boards…

      2. Jayme Snyder

        Also, remember that time we bought a bunch of PCIe flex cables and stuck as many NIC cards as we could in a single super micro 1u chassis with the lid off. Those were good times.

  16. AdiDOS

    Does anyone know if Riser will works with different Versions?
    Riser PCI-E x16 Bifurcation 2x PCI-E x8, one with PCI-E 3.0 x8 and one with PCI-E 2.0 x8.

  17. peter j connell

    I agree that lots of low bandwidth links (like a 8x pcie x1 slot or nvme m.2 slot adapter card) to the cost effective power of some mainstream peripherals, would be a powerful tool for some powerful apps.

    We see crypto miners buying ~custom mobos which maximise pcie x1 slots, & using each to host a ~massive GPU.

    Similarly, nvme seem a powerful entity when teamed for their largely hidden processing power.
    dunno, but apart from massive, near memory read speed storage space, they boast as much as 18 processors on the latest consumer nvme.

  18. Don perhaps you can steer me in the right direction. 2 x Samsung 1TB 970 Pro nvme M2 plus 2 x Samsung 2 TB Evo Plus nvme M2

    Onto my Gigabyte GA-Z77X-UP TH with i7-3770K Ivy Bridge and 16 Gb Ram running Windows 7 64 Ultimate. Seems to leave me one pcie 3 slot light.
    Bios is American Megatrends F12 15/5/2014
    SYMBIOS ver. 2.7
    Am trying to maximize best ssd quality with max TB i can stack on pcie.

    Love to get more on but i cant fathom SK Hynix site for ssd say 4tb or 8tb (in suitable module) . My sense is $$$ might preclude me anyway.
    Noticed you have bought hynix yourself. but that site loses me.let alone how to source a competitive seller or in fact any retail seller.

    Have installed microsoft hotfix for native nvme boot on W7 (off lenovo site) At least research seems to indicate for boot.
    It is a dual chip UEFI board (with 2 x Thunderbolt 1 ver. available externally unused).

    Bandwith is shared over 16x , 8x, and 4x slots
    If x 4 is populated then the x 16 operates at x 8 and x 8 then operates at x 4.
    So logically that leaves me trying to fit 2 of those drives on one x8 slot.
    Which i assume means bridged card for those tandem m2’s
    and the other 2 single m2 slots have in effect board instigated bifurcation covered.

    Could possibly help with some ideas or home truths on my goals and possibilities.
    Regards Peter b

    1. db

      so with the 3770k i wouldn’t sweat the pci bandwidth too much, the processor will top out.
      get a card w/ a bridge and then all will work.

      as for the hynix… i’m not convinced i would recommend it. It has the supercaps to give safe power down, and was economical, but is not very fast.
      i bought mine on ebay, they were new, but pulled from servers. So e.g. they were shipped w/ some OEM config and then parted out on receipt.
      Mine are still in service, they run ceph on an aging dual 2699 v3 system.

  19. Peter Black

    Much appreciated Don . That was very quick response.
    Any suggestions on the bridge card manufacturer or model.
    Am looking at buying now.
    Peter b

    1. db

      nothing to recommend. Its pretty generic by this stage of PCI life.

  20. Emsi

    I have X9DA7/E with two E5-2670 (40 lanes). I’ve installed Asus PCIe 3.0 x16 and enabled bifurcation 4x4x4x4 and no matter what I do with the card I can see only 1 disk (oddly enough usually from slot 4) or none at all. All other PCI-e slots for CPU1 are free (GPU is in CPU 2 link2 slot 5). I tried the card in Slot 1 (CPU 1 link 3) and Slot (CPU 1 link 2) but no luck. Could it be that the bifurcation is not working despite being available in the BIOS or it is a problem with the card?

    1. db

      i think its unlikely the bifurcation is not working, its actually just wires from the cpu to the slot.

      the card has no bridge, so its just wired nvme to slot.

      does this help:
      https://forums.servethehome.com/index.php?threads/smci-x9-bifurcation-i-why-some-boards-but-not-all.24223/page-3

      1. Emsi

        Thanks but unfortunately it doesn’t help. I can connect as many NVMes as I can stick into pcie ports (in fact I’ve tested 3) and all of them works but consumes a whole slot. However when it comes to enabling bifurcation and aforementioned card I experience inconsistent results where at most one of the drives is working (sometimes none) and the outcome changes with reboot (say I see the drive in slot 1 and after reboot I see the drive in slot 2).

Leave a Reply

Your email address will not be published. Required fields are marked *