I’ve been thinking a lot lately about a problem that is kind of obvious once you look at it, but somehow also just accepted as normal in modern operating systems: we spend an absurd amount of effort making sure applications cannot access memory they are not supposed to access, while at the same time giving the kernel access to basically everything.
Usually that is fine, because the kernel is trusted. But if the kernel gets compromised, most of those nice isolation boundaries suddenly stop meaning very much. If an attacker gets arbitrary kernel read/write, they can usually read application memory, modify it, steal secrets, patch processes and generally do whatever they want. At that point the distinction between “kernel compromise” and “full machine compromise” becomes mostly academic.
So I started asking myself a fairly simple question: what if the kernel just could not access application memory?
Not because of a permission bit. Not because the kernel is politely asked not to. I mean actually cannot, because the hardware architecture does not give it access in the first place.
That is basically the idea I want to research.
The current concept is to split the system into two separate processing domains: one Application Processor and one Kernel Processor. The Application Processor runs applications, the Kernel Processor runs the kernel, and both get their own private memory. Between them is a third memory region that both sides can access and that acts as the communication area.
So instead of having one CPU where userspace and kernel space are separated by privilege levels, you would have something more like this:
Application Processor
┌──────────────────────┐
│ Applications │
│ │
│ Private AP RAM │
└───────────┬──────────┘
│
Shared Memory
│
┌───────────┴──────────┐
│ Kernel Processor │
│ │
│ Private KP RAM │
└──────────────────────┘
The important part is that the Kernel Processor would not be able to map or directly access the Application Processor’s private memory. If the kernel gets completely compromised, arbitrary code execution and everything, it still should not be able to just point at an application address and start reading.
And yes, at that point syscalls basically become IPC. I am apparently reinventing message passing, except with extra hardware.
If an application wants the kernel to do something, it copies the data the kernel actually needs into shared memory, sends a request to the Kernel Processor and waits for the result. The kernel gets access to the data that was explicitly handed to it, but not automatically to the rest of the process memory.
For something simple like writing data, that might look roughly like this:
Application
│
│ copy data
▼
Shared Memory
│
│ syscall request
▼
Kernel Processor
The really interesting part is not the message passing itself. We already know how to do message passing. The interesting part is changing the trust model.
Right now the kernel basically says: “I manage your process, therefore I can see everything.”
In this architecture it would be closer to: “I manage resources for you, but I only get the data you explicitly give me.”
That sounds like a relatively small difference, but from a security perspective it is huge.
Of course, the first immediate problem is something like copy_from_user().
Linux, for example, constantly needs to move data between userspace and kernel space. You cannot realistically port a normal operating system and simply pretend that this does not exist. So one thing I want to explore is whether these operations could become hardware-mediated instead of being direct memory accesses.
The Application Processor could authorize a specific memory region for transfer, then the Kernel Processor could request that transfer through a separate memory controller or gateway. The hardware checks whether the requested region was actually authorized, and only then copies the data.
So instead of:
Kernel → user pointer → read whatever
you get something more like:
AP Memory
│
│ explicitly authorized
▼
Memory Gateway
│
▼
Shared Memory
│
▼
Kernel Processor
The Kernel Processor never needs direct access to AP memory at all.
If this works properly, even a fully compromised kernel would still be stuck behind that boundary. It could request transfers, but it could not simply decide that some random application address suddenly belongs to it.
That is basically the security property I care about.
Obviously this does not magically make a compromised kernel harmless. If the Kernel Processor is completely owned, the attacker can still do a lot of damage. They can probably break scheduling, interfere with networking, return fake filesystem data, deny services, mess with devices and generally make the machine unusable.
But ideally they cannot read arbitrary application memory.
For example, imagine an application has encryption keys sitting in its private memory. Today, if the kernel gets arbitrary read access, those keys are basically gone. In the split model, compromising the kernel alone would not be enough. The attacker would additionally need to break the AP/KP isolation mechanism itself.
That means the attack surface moves away from the entire kernel and towards a much smaller number of components: the shared memory protocol, mailbox interface, external MMU, memory transfer controller and whatever mechanism controls the separation between AP and KP memory.
Which, yes, technically means I am solving one large security problem by inventing several smaller security problems.
But that is actually the point.
A modern kernel is enormous. Linux is tens of millions of lines of code, contains drivers for basically every piece of hardware humanity has ever produced and has decades of historical behaviour that cannot simply disappear because someone had a nice architecture idea.
A memory gateway, on the other hand, could potentially be tiny.
The less code and hardware logic involved in enforcing the critical security boundary, the easier it becomes to reason about whether that boundary is actually correct.
One thing that definitely would not be enough is simply running the kernel on a separate CPU core. If both processors still share the same physical memory, then the Kernel Processor can probably still access application RAM somehow. At that point we have made the architecture more complicated without actually gaining the security property I want.
The memory isolation has to exist below the kernel.
The Kernel Processor should ideally not even be physically capable of generating a valid memory transaction targeting AP RAM. It should not matter whether the kernel is running normally, has a bug, or is executing an attacker-controlled payload. The hardware should just reject it.
Basically:
KP: can I read this AP address?
Hardware: no.
KP: but I'm the kernel.
Hardware: still no.
That is the experiment.
I am not trying to redesign Linux immediately either. That would be a great way to spend six months debugging completely unrelated kernel behaviour and learn absolutely nothing about whether the architecture itself works.
The first prototype will probably be built in QEMU with a very small custom kernel and a very small application environment. The only thing I really need in the beginning is one simulated Application Processor, one Kernel Processor, separate RAM regions, shared memory and some kind of mailbox or interrupt mechanism between them.
The first successful demo can be something incredibly advanced like printing “Hello World”.
The AP puts the string into shared memory, sends a request to the KP, and the KP prints it. Congratulations, several weeks of architecture work and we have successfully recreated a computer from the 1980s.
After that comes the actually useful test: intentionally compromise the Kernel Processor.
Give it arbitrary code execution. Let it read and write all of its own memory. Let it completely control its own kernel environment.
Then put some secret value somewhere in AP-private RAM and see whether the KP can retrieve it.
If the answer is still no, then the architecture is doing something interesting.
There are obviously a lot of ugly questions that come after that.
Performance is probably one of the biggest ones. Modern syscalls are fast, and turning memory accesses into explicit transfers between processors is not going to be free. Large I/O operations could become especially expensive if everything needs to bounce through shared memory.
Then there are things like page faults, process creation, signals, debugging, shared libraries, memory-mapped files, shared memory between applications and all the other features normal operating systems quietly depend on.
DMA will also be fun.
There is not much point carefully isolating AP RAM from the kernel if some PCIe device can simply DMA straight into it and completely ignore the entire architecture. So eventually the same isolation model would need to apply to devices as well, probably through an IOMMU or a similar hardware layer.
There is also a deeper question around what the kernel should even be responsible for in this model.
The kernel currently enforces a lot of security policy because it has full visibility and control over processes. If applications become more opaque to it, some responsibilities might have to move into the Application Processor side or into a separate trusted component.
So I do not expect this architecture to be simple. I also do not expect the first version to be particularly fast.
The thing I actually want to find out is much narrower:
How much authority does the kernel really need?
We have spent decades protecting the kernel from applications. Maybe it is worth also thinking about protecting applications from the kernel.
The kernel can still schedule processes, manage devices, provide filesystems, networking and system services. It just does not necessarily need unrestricted access to every byte of memory belonging to everything running on the machine.
If that boundary can be enforced in hardware, then “kernel compromise” might stop automatically meaning “everything is compromised”.
And if it turns out this entire idea is horribly impractical, at least I should end up with a very complicated QEMU project and a better understanding of why computers are built the way they are today.