Product News
March 15, 2019
MuSE: Clustering, Extensibility & Almost Infinite I/O Power with OPAL-RT Simulations
OPAL-RT has improved the user experience when using multiple simulators, remote targets or when requiring clustered machines for increased I/O or other capabilities. This high-speed link is referred to as MuSE (Multi-System Expansion link).
Irène Pérès, MuSE’s Product Owner at OPAL-RT, recently joined us to speak about the incredible power, flexibility, versatility and user-friendliness of this special add-on—that is so much more than just an expansion kit.
Interviewer (IV): Thanks for joining us, Irène! I have a partial understanding that MuSE is sort of a hardware expansion kit, allowing clustering of multiple machines, and frequently used for augmenting available I/O. Is that how you’d describe it?
Irène Pérès (IP): “MuSE does indeed involve some hardware components: optical fiber and SFP (Small Form-factor Pluggable) transceivers. However, the hardware is just the tip of the iceberg, as the essence of MuSE lies in the software layers we’ve developed. The problem we needed to solve was: when a user needs more I/O channels than are available in one chassis, how can we facilitate connection with other chassis to add that additional I/O capacity?”
IV: MuSE has been used for simulation in industries where they tend to use a lot of I/O, like aviation and aeronautics—is that right?
IP: “Yes! We first developed the link with these industries in mind. But actually, we see an emerging trend to use MuSE in very diverse applications and even for smaller systems. The user-friendliness of the modular joining of the systems is a very persuasive selling point for its usability. It makes things a great deal easier. So, for example, if you have an OP4510 simulator with its 128 I/O channels and need more I/O, with PCIe you could add an OP4520 expansion chassis and double the capacity to 256 I/Os–but what if you needed even more? The OP4510 is a fairly small box, so it can’t take more than one PCIe card. But now, using the four SFP ports in the front of the chassis, we can connect four I/O expansion chassis to that OP4510, and these chassis can be of any type–even bigger ones like the OP5607 with 256 I/O channels, increasing the I/O capability exponentially.”
IV: So you mean you can mix any type of OPAL-RT chassis with MuSE?
IP: “Indeed! As I mentioned, we were using SFPs to connect to third-party devices (amplifiers, MMC controllers), and already had SFP ports on all our newer systems. So the software we developed is compatible with all systems, which allows the user to build a network of simulators and expansion chassis of different types, depending on their requirements and budget. On our OP5707 simulator, for example, we have 16 SFP ports, so we can connect 16 OP5607 chassis, each with up to 256 I/Os. We have even recently upgraded the OP5600 family to add MuSE capability, with the new OP5650 just released, which has an Artix-7 FPGA and 4 SFP ports.”
IV: So really: up to 4,096 I/O connections? Have we ever done anything like that as far as you know?
IP: “Well, let’s run through some pretty exceptional simulation challenges, and it’ll give you an idea of the flexibility and expandability we can provide. Configurations like this simply weren’t possible with PCIe in a user-friendly way, if at all:
- One of our clients needed to expand their OP4510 simulator (running with RT-LAB) to perform real-time control of power electronics and electrical machines on a test bench. In that case, the addition of four OP4200s clustered allowed them to run the control on a real-time calculator of the OP4510 while piloting the inverters and collecting data from measurements on the OP4200s;
- Another client has two setups, one OP5707 with four OP4520s, and another OP5707 with four OP4200s, running with HYPERSIM. In this case, the remote units are located ~100 meters from the simulator, and the customer needs to use the optical fiber pre-installed in their building. We could not add real-time synchronization cables between the units, as we had done previously, and with this setup in mind, we integrated our real-time synchronization signal into the MuSE link, so that all communications are now over one single link.
- Finally, one of our long term clients had bought several of our chassis (OP5600, OP4520, OP5707, OP4510) and wanted to reuse these for a larger project they were preparing. Adding the MuSE capability allows them to connect their OP4510s and OP4520s to the OP5707 to collect data from measurements, yet still be able to easily disconnect an OP4510 to use it as standalone chassis in other experiments when needed.”
IV: Well we’ve talked a lot about hardware, but you mentioned the software layer was a large part of the link. What exactly does it bring, in term of expandability?
IP: “Usability, both during model preparation and during execution of the simulation, was our main objective. In the past, we had developed a generic Xilinx Aurora interface that users could use to prepare FPGA bitstreams for our chassis. But managing the data packing/unpacking and the timing constraints of the protocol was cumbersome, and we understood that our users were obviously more interested in developing their control algorithms than tackling these low-level problems. Now the Aurora integration in the bitstream is completely taken care of by our RT-XSG toolbox during the generation of the bitstream.”
“And for the CPU model, we made it easy through the RT-LAB and HYPERSIM interfaces: the user has only to add remote units through the GUIs. The simulator’s bitstream will not need to be changed if the users need to use one more remote unit later on —everything is taken care of. Much more user-friendly, and a huge step forward!”
IV: Pretty exceptional, and we helped them all out. Okay, so what we refer to as MEA (More Electric Aircraft), or electric boats, for example–it’s not hard to imagine a scenario where we actually need 4,056 connectors! Given the increasing electrical/electronic complexity of some of the transportation and other electronic networks and infrastructures out there, that is.
IP: “We know, through the installations I’ve outlined to you above, that these setups work reliably. Knowing this and going forward, in terms of expandability, it’s more an issue of time, of latency—how much data can you transfer in a time-step? But you have the same considerations as PCIe. We start to hit the ceiling at some point–but that’s not a limitation distinct to MuSE, or imposed by OPAL-RT.”
IV: And in term of performances, do the optical fibers reduce latency?
IP: “Actually it is not the link itself which removes latency, but the fact that we no longer need to use one or more PCIe slots managed by the chassis’ motherboard, and different PCIe interconnection cards and cables with various chassis. We thus gained better control by using the Aurora link, and the only PCIe connection remaining is between the simulator’s motherboard and the simulator’s internal FPGA. The rest of the communication happens between the FPGAs of the various chassis directly. We also control the network enumeration, the detection of the number and type of chassis connected, etc.”
IV: In your experience, is this a highly specialized configuration for a certain type of user…?
IP: “Not at all, perhaps even the opposite. Before, when we had PCIe connections, the users didn’t have to worry about it, the motherboard did. So we wanted to have the same ease of use with the SFP and optical fibers. Generally, all the user has to do is cable the SFPs, and configure the simulator and its remote chassis in RT-LAB or HYPERSIM. The way the users interact in the model, it’s the same as with any other chassis–there’s nothing special to program.”
IV: So I understand this doesn’t just increase the I/O of the machine, but it also allows many other connection schemes. So this is not just a linear improvement in use, but an exponential one. Are there other developments coming?
IP: Of course! We achieved ease of use for the CPU model and the bitstream preparation. But we are now tackling other aspects of FPGA to FPGA communications required by our users: models distributed over multiple FPGAs, communication with power amplifiers, with MMC controllers, etc. I think we’re just in the first phase of this MuSE project, and there are plenty of interesting features still to come!
About the Interviewee
Irène Pérès joined OPAL-RT in 2001, and she has been involved in the development of OPAL-RT simulators, including software drivers, firmware and aspects of hardware management.
She is now a Product Manager and technical fellow and focuses on the evolution of OPAL-RT multi-FPGA platforms.
From her early career years (she received a Ph.D. in Plasma Physics from Paul Sabatier University–in Toulouse, France–in 1990), she retains a strong interest in challenging, complex technical projects, and strives to bring simplicity to these challenges.
