I know what I’m doing with that title. If you got the reference, welcome cultured people.

The idea

I was reading textbooks for the NoC when I came across an interesting combination of words: dimension-sliced xbar (xbar is slang for crossbar).

The more I thought about it, the more I realised that with XY routing, some connections in the crossbar are never touched. A packet finishes its X movement before it starts moving in Y, so some input-to-output combinations can never happen.

A 5×5 crossbar has 25 possible connections, but only 16 of them are legal under XY routing. The other 9 are things like U-turns and reverse-dimension turns that a correctly routed packet should never request.

So naturally I thought: why am I building hardware for connections that can never happen?

That was the idea behind the dimension-sliced xbar. Instead of one full 5×5 crossbar, split the switch fabric into two smaller ones, one for each routing dimension, with a bridge between them. Roughly:

• X crossbar: East, West and Local-side inputs, with outputs for continuing in X, moving toward the Y side, and local ejection.
• Y crossbar: North, South, Local-side input and the bridge input, with outputs for North, South and local ejection.
• The Local input can participate in both dimensions, so the final local ejection paths have to be handled carefully.

The X side handles X-phase traffic, the Y side handles Y-phase traffic, and the bridge lets a packet move from one to the other.

It looked reasonable. The crossbar takes up a substantial amount of space in a NoC, so removing hardware I already knew would never be used felt like a pretty good optimisation.

Then I decided to actually measure how much I was gaining. Well, let’s just say it was not good.

So I decided to measure it

This was also a good chance to understand Yosys and ABC properly, instead of treating synthesis as a command that gives me a number while I pretend I know what happened.

I wrote the logic for both the full 5×5 xbar and the dimension-sliced version (xbar is slang for crossbar), and ran the experiments.

I started with the generic LUT mapping flow in Yosys, using 4-input LUTs:

I looked at that and thought:

I am intelligent.

The dimension-sliced version was significantly smaller. This was going exactly how I imagined it would.

So instead of stopping there like a responsible engineer, I took both designs to Vivado.

Then Vivado happened

For the FPGA experiment I targeted xc7a200t-fbg676-2 with Vivado 2025.2. The result:

Wait. 320? Both?

I had just seen 1125 versus 711 in Yosys, and now Vivado was telling me the two designs were identical in size. I had no idea what was happening. This was the point where I stopped pretending I understood synthesis.

So I went back to Yosys

This time I didn’t only run the 4-LUT experiment. I swept the LUT input width:

And I sat there for a while.

The LUT input count itself changes the result. It isn’t just design → number of LUTs. It’s design + target LUT architecture → number of LUTs. The dimension-sliced design wins at the smaller LUT widths in this generic experiment, but once the LUTs get wide enough the full crossbar becomes smaller.

That explained why my first LUT4 number wasn’t some universal truth about the two architectures. It was a result for one particular LUT model, and it was far more useful than the number that made me feel clever.

And then I looked at timing

I ran both designs through Vivado timing analysis with a virtual 10 ns clock. Placement was not constrained and neither design met the target, so these are not final NoC timing numbers. I mainly wanted the relative comparison.

The full crossbar had the better WNS and TNS.

My best guess is the bridge. I added it to make the architecture smarter, but a path from an X-direction input to a Y-direction output now has to go through both sides of the split, which adds a level of logic to the common turning case. So I removed some impossible connections and then added a physical bridge that made the turns that actually happen slower.

That hurt.

At this point I was ready to blame FPGAs. Vendor locked. Corporate. Weird architecture. Yosys, on the other hand, was sitting there being open source and letting me poke around inside the flow, and I love open source.

Maybe the FPGA world just wasn’t ready for my intelligence yet. Maybe I was destined for the ASIC world.

ASIC world, here I come.

Then the ASIC experiment put the final nail in the coffin

I took both designs and mapped them to the Nangate45 standard-cell library. Now I wasn’t thinking in LUTs anymore. The design gets mapped into actual standard cells, and I was suddenly looking at things like AOI21_X1, AOI222_X1, AOI22_X1 and OAI33_X1.

What are these?

AOI and OAI cells combine several Boolean operations into a single cell. Instead of building everything from tiny primitive gates, the mapper can recognise a Boolean structure and map it onto a cell that already implements it. I had spent a lot of time thinking about muxes and LUTs, and this was a completely different way of looking at the same RTL.

Then I got the area report.

The full crossbar was 61.446 µm² smaller, about 7.49%, than the version I had specifically designed to remove unnecessary hardware.

The full crossbar mapped to:

The full crossbar still had all those theoretically illegal connections. I had deliberately removed them in the other design. And the full crossbar still came out smaller.

I was dumbstruck.

The worst part wasn’t that the full crossbar won. It was that I didn’t understand why. I could explain the FPGA result, because I could look at the LUT architecture and see that the target primitive changed the answer. Here I had a standard-cell library, no LUT-width ceiling to blame, and I had still made my supposedly optimised design bigger.

That was honestly more painful than a bad number. It was also the part that made this experiment worth doing.

The thing I realised about ASIC and FPGA synthesis

This slowly made me realise that FPGA and ASIC synthesis really are two different ball games.

In college, we learn an ASIC-esque design flow: write RTL, synthesise, talk about timing and area, and then very often the story ends with run it on an FPGA and call it a day. I don’t mean that as an attack on what we learn. It’s a useful flow and it gives you something physical to work with.

But there is a whole other world behind it. The FPGA flow asks you to map your logic onto a specific set of programmable resources. The ASIC flow hands the tool a standard-cell library and lets it choose from a completely different collection of building blocks, with different drive strengths, cell areas, and timing characterised for that library.

That was probably the most valuable part of the whole experiment.

So was the dimension-sliced crossbar actually bad?

For the Xilinx target I tested, the dimension-sliced design gave no area advantage and made timing worse. For Nangate45, the full crossbar was about 7.5% smaller. At first glance the conclusion is obvious: dimension slicing was a bad idea.

Except I don’t think that’s quite right.

While writing this up I went back through the results again, and the conclusion I was originally writing towards was slightly wrong. At 4 LUT inputs, the dimension-sliced design was 711 LUTs against 1125 for the full crossbar. That’s not a small gap. If I were targeting a technology where 4-input LUTs are the real primitive, the idea suddenly becomes worth discussing.

That doesn’t mean every 4-LUT FPGA will love my architecture. The Yosys result is still a generic mapping experiment. But it does mean the idea wasn’t stupid. I was asking a technology-dependent question and expecting a universal answer, and synthesis doesn’t work like that.

What I actually learned

I started this because I thought I had found an obvious optimisation. Nine connections that XY routing never uses. Remove them, smaller hardware, done.

Instead I learned that:

• a generic LUT experiment is not the same as mapping onto a real FPGA,
• LUT input width can completely change the result,
• FPGA and ASIC synthesis optimise for very different target architectures,
• an architectural reduction does not automatically mean a physical area reduction,
• and the synthesis result can be more interesting than the architecture idea that started it.

What I haven’t tested yet

Looking back at it, I actually tested two ideas at once and only noticed afterwards.

The first was pruning: remove the 9 connections XY routing never uses. The second was the structure I chose to implement it, which was two smaller crossbars with a bridge between them. Every result I got is for the combination. The bridge is my best explanation for the timing loss, and the full crossbar collapsing to exactly 160 LUTs at wide LUT widths suggests it maps very cleanly, while the two-stage version never gets there.

So I can’t say pruning is a bad idea. I can only say pruning with a bridge lost on the targets I tried.

The experiment I haven’t run yet is the obvious one: keep a single-level crossbar and just delete the 9 illegal inputs from each output mux. No slicing, no bridge. If that one wins, the idea was fine and my implementation of it was the problem. If it loses too, then I’ll finally have a result that points at the idea itself.

And underneath that is the bigger thing this left me with. I want to put FPGA and ASIC synthesis side by side and understand why the same RTL changes shape when the target changes, not just read off a LUT count.

I’ll write it up when I get to it.

Pick your poison.

It all started with a simple question:

How many body flits is too many body flits?

Is having 100 body flits in a packet okay? Sure, but then the resources associated with that flow remain occupied for a brutal amount of time.

It’s crazy to get all that transmission efficiency from sending just one head flit and one tail flit, only for a small packet to be trapped behind it waiting for its turn.

This is called serialization delay.

So the naive man thought, let’s improve fairness in the arbiter.

But alas! Fairness doesn’t make the packet shorter. The arbiter can be perfectly fair, and the Virtual Channel (VC) is still held hostage while all those body flits slowly make their way through.

The naive man was naive again.

A 100-flit packet simply consumes more cycles, and in a wormhole-style design, the VC allocation remains locked until the tail flit finally passes. At first glance, this looks like a purely temporal problem: the packet simply exists for too long.

The obvious solution is to give the router more choices. If one VC is occupied, let another packet use a different VC, assuming the required downstream buffers are available.

“Then increase the VCs,” he said.
“But SRAM is costly,” they said.

This is capitalism baring its fangs.

And then the arbitration logic bites back, too. Every additional VC adds another possible request that the allocator has to consider. More VCs mean more state, and wider, far more complex allocation logic. The exact cost depends on the router architecture, but eventually, the hardware needed just to decide who gets the resource starts becoming part of the bottleneck itself.

Which makes this feel like a temporal problem disguising a spatial one.

We started by worrying about how much time a packet spends holding a resource. But the only way to escape that temporal delay is by spending physical hardware: more buffers, more VCs, more arbitration gates. The delay is temporal, but the way we buy concurrency is spatial. And the cost of that solution is spatial, too.

So perhaps the real problem was never just that the packet takes too many cycles. It is that we have a finite amount of hardware trying to force temporal contention into spatial concurrency.

We have gone in a full circle, and the original question returns:

How many body flits is too many?

Apparently, there is no magic number.

Just a collection of tradeoffs waiting to bite you.

I wanted to tackle the FIFO first for two reasons. First, I was confident I could write one. Second, it is the first block the data hits when it enters the router, so it made sense to start there.

I was thinking I didn’t want this to be a simple FIFO. Everything from virtual channels to contention eventually rides on these buffers, and these little simple shits are expensive. SRAM is expensive, so I wanted to make sure I was getting as much use out of it as I could.

A simple FIFO is pretty straightforward. You have a memory, a read pointer, a write pointer, and some logic to tell you when the FIFO is full or empty. That would have worked, but I wanted to see if there was anything useful I could squeeze out of it before calling it done.

I already knew about FWFT FIFOs, and I liked the idea behind them. If the data is already available, why make the consumer wait another cycle just because the FIFO wants to be formal about the procedure?

I didn’t need a full FWFT implementation for the router, but I could take the idea and apply it where it actually helped. So I added simultaneous read/write handling and an empty-FIFO bypass.

If the FIFO is empty and a read and write happen in the same cycle, the incoming data can go straight to the output instead of taking a trip through the memory just to immediately come back out.

That felt like a useful little optimisation rather than adding complexity for the sake of it.

Then there was the other side of the FIFO, the part that deals with the router’s flow control.

The router uses a credit-based system, so a write should only happen when there is space available in the buffer. I kept the credit logic outside the FIFO itself, but added a protocol error at the FIFO level when an illegal operation is attempted. A write while full and a read while empty both raise err. It’s mostly there as a debugging signal, but I like having the FIFO complain when something that should never happen actually happens.

There is another case I wanted to make explicit too. If the FIFO is full and both read and write are asserted, the read still happens but the write is rejected. The FIFO was full when the write was presented, so I don’t want the behavior to depend on treating the simultaneous read as some kind of retroactive permission.

I liked the design I ended up with. I think I hit a decent sweet spot where I didn’t over-engineer the FIFO, but I also didn’t just grab an off-the-shelf FIFO and call it a day.

Then, while writing this post, I came across something.

Wait. How do I handle credits for the bypass condition?

The flit never actually enters the FIFO in that case. And there is the full-FIFO case too, where the read happens but the write is rejected. The FIFO’s occupancy and the interface signals aren’t quite telling the same story anymore, so the credit controller needs to account for what actually happened.

I think I can handle both cases using the full/empty conditions, the read and write pointers, and some combinational logic to track these states.

Hmm.

The blog is useful after all.

Now there is another voice calling me, and this one is asking for numbers.

The RTL tells me what the FIFO does. Simulation can tell me what it actually does under traffic.

I can put some instrumentation behind an `ifdef SIM and start collecting things like buffer occupancy, how often the bypass is actually hit, and how long a flit spends waiting in the buffer.

Now that is gold.

I want to see those numbers.

Five-port router in the 2D mesh. .

Router microarchitecture.

Its job is deceptively simple: accept a flit, figure out where it needs to go, acquire the resources required to get there, resolve contention, and move it to the next router.

The difficulty is that all of those things are competing for the same few cycles.

The current router uses a 5-port input-buffered architecture with two virtual channels per port, credit-based flow control, deterministic XY routing, and a pipelined datapath:

BW → VA → SA → ST → LT

The stages are arranged so that work which can be done early is done early, while the stages responsible for actually moving data are kept as clean as possible.

Starting with the Buffer

The first architectural question was where to put the buffers.

Input buffering is the obvious answer only until you stop and ask why.

A flit arrives at an input port, but it may not be able to move immediately. The router may be busy, the required downstream VC may not be available, or the flit may lose switch arbitration. Something has to hold it while that gets sorted out.

With input buffering, that waiting happens exactly where the flit arrived.

The flit sits in the input FIFO until it has everything it needs to move. The input side does not need to know whether the crossbar is available or whether the flit will win arbitration. It just needs to accept the transfer when it has buffer space.

That makes the input buffer a useful boundary between receiving traffic and deciding what to do with it.

Why not output buffering?

The alternative initially seems reasonable.

Suppose North and West both want East in the same cycle. Only one flit can physically leave through East, so one of them has to wait somewhere. An output buffer seems like a natural place to absorb that contention. It appears to free up the input side and let incoming traffic continue while the output handles the waiting.

My first assumption was that this would then require another arbitration stage at the output to decide which waiting flit should be served next.

Then I looked at the router pipeline again.

SA has already made that decision.

Switch allocation has already decided which input gets access to a given output for the current cycle. The crossbar is not making another scheduling decision. It is simply implementing the connection selected by the allocator.

So an additional output arbiter would just be making the same decision twice.

That still leaves the more fundamental question: what happens if the input buffers disappear entirely and only the output side is buffered?

Now the crossbar has to accept traffic from every input as it arrives, with nothing upstream holding a flit while the router decides whether it can proceed.

There are two obvious ways to make that work.

The first is to run the crossbar faster than the rest of the router so it can drain all possible incoming transfers within a single router cycle.

No.

The second is to buffer at the crosspoints of the crossbar. For an (N)-port router, that potentially means (N^2) buffering locations.

Five ports is not catastrophic. The problem is the direction of the scaling. The buffering requirement grows quadratically with the number of ports.

Multi-write-port FIFOs can change the implementation tradeoff, but introducing more complicated memory structures simply to avoid buffering at the inputs is difficult to justify.

Input buffering therefore remains the architectural choice.

Not because it was the first option considered, but because the alternatives were followed far enough to expose their actual costs.

The Flit Lifecycle

With the buffering decision made, the next question is what actually happens to a flit once it enters the router.

At a high level:

Buffer → Allocate VC → Allocate Switch → Traverse Crossbar → Traverse Link

The actual pipeline is:

BW → VA → SA → ST → LT

Each stage has a fairly specific job.

BW: Buffer Write

A flit arrives at an input port.

The first thing that happens is, thankfully, not routing.

The incoming transfer is sampled on the clock edge and the flit is written into the input FIFO. This stage is deliberately simple. No routing, no arbitration, no credit management.

Just:

Is the transfer valid? If it is, write the flit.

I want the rest of the router to work with registered state rather than whatever happens to be sitting on the input wires. There is no point doing interesting work before the data has even been captured.

Once the flit is in the input buffer, the router can start working on it.

Route Computation

If the flit is a head flit, route computation begins.

The slightly strange part is that the router is calculating the route for the next router, not this one.

This router already knows where the flit is. What matters is where it needs to go after it leaves.

The destination information is extracted from the header and used to determine the next direction. Since the network uses deterministic XY routing, the decision is straightforward, but the important part is when it happens.

The route is computed early rather than becoming another piece of work on the critical path when the flit is finally ready to cross the switch.

The result can sit there while the flit waits for the resources it needs.

Might as well make the waiting time useful.

VA: Virtual Channel Allocation

The head flit now needs a VC in the next router.

Body and tail flits do not need to repeat this process. They follow the VC established by their head flit.

The mesh topology gives us a useful shortcut here.

If a flit needs to enter the West port of the next router, it has to leave the current router through its East output. The same relationship exists for every direction. The current output directly corresponds to the input port that the flit will use at the next router.

That means the VC request can be handled using the VC arbiter associated with the current output port.

If multiple input ports want VCs at the same downstream port, they are already competing at the same output here. The VC arbiter resolves that contention and grants one of the available downstream VCs.

This is where VA and SA need to be kept separate.

VA is asking:

“Can I reserve a VC at the next router?”

SA is asking:

“Can I use this router’s crossbar?”

Two different resources, two different decisions.

Winning VC allocation means the flit has secured a place in the next router.

It still hasn’t earned the right to leave this one.

SA: Switch Allocation

Now the flit reaches switch allocation.

There are two rounds because there are two different contention problems.

First stage

An input port can have multiple VCs with eligible flits.

Only one of them can be presented to the crossbar in a cycle, so the first stage chooses which VC gets to compete.

Now every input has at most one contender.

Second stage

The remaining contenders compete for the output ports they want.

North and West might both want East this cycle.

East is, unfortunately, still only one East.

The second stage resolves that contention and selects which input gets the output.

Nothing has moved yet.

VA and SA are making decisions. The flit is still sitting in the input FIFO.

If it loses arbitration, it stays there and tries again on the next cycle.

There is no partial credit for almost winning.

ST: Switch Traversal

Once the flit wins switch allocation, the router finally gets to move some data.

The crossbar establishes the connection selected by the allocator, and the flit traverses the crossbar to its granted output.

This is where all the earlier decisions become an actual data movement.

The route information computed earlier is also inserted into the flit’s payload at this point.

Until now, the route computation was local control information. Now it becomes part of the flit itself and travels with it to the next router.

The router has done its thinking.

The flit gets to leave.

LT: Link Traversal

The flit then traverses the physical link to the next router.

This is probably the least intellectually exciting stage. After all the allocation and arbitration, it is essentially:

send the flit.

But the work done in the previous stages matters here.

When the flit arrives at the next router, the route information carried in its payload is already available. The next router does not need to spend another cycle figuring out the direction from scratch.

It can use the information computed by the previous router and continue the same process.

Then the cycle repeats.

Virtual Channels and Flow Control

The router has two virtual channels per port, with credit-based flow control between neighboring routers.

The important distinction is that a virtual channel is not another physical link. Multiple logical channels share the same physical resources while maintaining separate flow-control state.

The credits provide the information needed to determine whether the downstream buffer has space before sending a flit.

This fits naturally with the input-buffered architecture. The downstream router owns the storage, while the upstream router keeps track of whether that storage can accept another flit.

The resulting relationship is simple:

Downstream buffer space → credit → upstream transmission permission

The details of the credit state and per-VC control form part of the input-port architecture rather than being mixed into the data path itself.

The Crossbar

The crossbar is the physical structure that connects the router’s inputs to its outputs.

It does not decide which input should win.

That decision has already been made by switch allocation.

This distinction matters because it keeps the control and datapath separate. SA produces the connections that are allowed for the current cycle, and the crossbar implements those connections.

The result is a clean division:

Allocator: decide.

Crossbar: connect.

The crossbar therefore does not need to understand packet destinations, VC ownership, or arbitration policy. It only needs to establish the selected input-to-output paths and move the corresponding flits.

Putting the Router Together

At this point the router can be viewed as a sequence of increasingly constrained decisions.

A flit first becomes local state.

BW

Then its destination is determined and, for a head flit, a downstream VC is requested.

VA

Once the required resources are available, it competes for the crossbar.

SA

If it wins, the datapath finally moves it.

ST

Then the physical link carries it to the next router.

LT

The entire path is therefore:

              ┌───────────────┐
              │  Input FIFO   │
              └───────┬───────┘
                      │
                      ▼
                     BW
                      │
                      ▼
                     VA
                      │
                      ▼
                     SA
                      │
                      ▼
                  Crossbar
                      │
                      ▼
                     ST
                      │
                      ▼
                     LT
                      │
                      ▼
                Next Router

The architecture is intentionally structured so that eligibility, allocation, and movement are separate problems.

A flit can be present without being eligible.

It can be eligible without winning arbitration.

It can win arbitration without having physically moved yet.

Only after all of those conditions are satisfied does the crossbar actually carry the data.

That separation is what allows the router to pipeline the work rather than making every stage wait for the entire decision process to finish.

What Comes Next

The microarchitecture is now defined well enough that the next problem is no longer deciding what the router should be.

It is building it.

The immediate work is around the individual structures that make up this pipeline: the input-port cluster, VC control, arbitration, crossbar, and the ST/LT datapath.

Once those pieces are implemented and verified independently, they can be brought together into the complete router.

At that point the interesting question changes from:

“Can this router move a flit?”

to:

“What happens when an entire network of these routers starts moving them at the same time?”

That is where contention stops being a local issue and becomes a network-wide behavior.

If Project Linnea eventually becomes a working multicore system, that’ll be nice too.

I’ve started and abandoned enough half-finished write-ups to know how this usually goes. A burst of motivation, a post explaining what I’m about to build, and then silence for six months while the actual building happens somewhere no one can see it. The blog dies quietly. The project, if I’m lucky, doesn’t.

I’d like to try doing both at once this time.

Not because I think anyone is waiting for progress updates on a NoC from a final-year student in Hyderabad. I don’t think that. I’m writing this mostly for the version of me six months from now, who will have forgotten why a particular arbitration scheme seemed like a good idea at 2 AM, or what exactly broke during verification and how I fixed it. Notes fade. This is meant to be a slightly more permanent notebook than the ones I keep losing.

If someone else finds these notes useful along the way, that’s a nice bonus.

There’s also a simpler reason. I think better when I have to explain something, and I explain things better when I know someone else might actually read it. Most of what happens while building Linnea stays in my head, half-formed, until I’m forced to put it into words. This blog is where I force myself to finish those thoughts instead of letting them remain vague ideas that never quite become anything.

So this is what the blog is for: the good, the bad, and the ugly. Design decisions I’m proud of and others I’ll almost certainly regret. Things that work, things that don’t, and the occasional rant when something takes four days longer than it should have. If you’re here for polished updates only, you’re in the wrong place.

No promises on a schedule. Updated when I have something to say, same as the rest of the site.

Let’s see how far past three posts this gets.