Dimensional Multiplexing

Reconstructing weights could reduce what a model needs to store.

Dimensional Multiplexing starts with a practical constraint: a model can spend more time moving weights from memory than computing with them. The idea is to store a smaller shared set of weights and reconstruct the weights needed for each operation. If those weights share enough structure, additional computation could reduce storage without sacrificing the model's ability to perform the task.

The construction does produce distinct weights from shared stored values. Its storage costs can be counted exactly, including the transforms used for reconstruction, and it can be combined with quantization: representing weights with fewer bits per value. On one balanced synthetic composition task, it matched the performance of a dense INT4 model that required about 4.1 times as much storage. Here, dense means the model stores its weights individually, and INT4 uses four-bit weight values.

That task-specific result did not extend to language models. Across controlled transformer tests, compression of pretrained models and mechanism probes, reconstructing more weights did not outperform conventional models at the same storage budget. This suggests that performance in the tested construction depends more on the independently adjustable information it stores than on the number of weights it reconstructs.

A smaller stored base, expanded at use.

A logical weight plane is a set of weights reconstructed from a shared base. An address selects the transform that produces each plane. An optional residual stores a local correction, so a plane can differ in ways the shared base and transform cannot express. Here, q selects a plane and j indexes the shared base:

Suppose two bases are used to reconstruct eight planes, as shown above. Changing the address produces a different view of the stored information. It does not, by itself, add independently learned information. Adding a stored correction gives a plane more freedom, but also increases the storage cost. That is the tradeoff the experiments need to measure.

The controlled transformer changes only the feed-forward network (FFN), the part of each layer that transforms individual token representations through two weight matrices and a GELU activation. Attention, embeddings, positional encoding (RoPE), normalisation (RMSNorm), residual paths and the output language head remain conventional. This isolates the effect of replacing the FFN weights.

The tested version uses eight planes, all active on every forward pass, derived from two shared bases. Learned Givens butterflies mix coordinates through structured sequences of pairwise rotations. Optional two-sided low-rank residuals encode local corrections with compact matrix factors. Quantization-aware training (QAT) accounts for four-bit weights during training.

The language tests did not show an advantage at equal storage.

The primary comparison asks whether dimensional reconstruction predicts language better than a dense INT4 model at the same whole-model storage budget. The dense frontier is the measured relationship between storage and prediction loss across the dense controls. The storage multiplier compares the dense storage needed to match a dimensional model's performance with that dimensional model's storage. A value above 1.0x would indicate an advantage.

Training budget also matters. Matching the total number of training tokens and matching tokens per stored parameter answer different questions, because models with different stored parameter counts receive different total training in the second comparison. Those comparisons are kept separate. Test loss is negative log-likelihood (NLL), which measures how poorly the model predicts held-out text; lower is better. The first table reports loss in nats, using natural logarithms.

Controlled transformer at the 20-million-parameter scale

FFN 1024 and FFN 216 denote the widths of the dense feed-forward networks. The dimensional configuration allocates a 10% residual uniformly, giving its reconstructed weights some local freedom.

The dimensional configuration had 0.02995 nats higher loss than the storage-matched dense control, despite using 0.5% more storage and doing 54% more arithmetic. In the separate comparison at matched total training tokens, its loss was 0.01752 nats higher. Both comparisons favour storing a narrower dense FFN over reconstructing a wider one from shared weights.

At the 50-million-parameter scale, the closely matched equal-byte pair remained unresolved. The storage multiplier could only be bounded at no more than 0.996x because the result fell outside the measured dense frontier. This is a bound, not an exact estimate, and it provides no evidence of an advantage emerging with scale.

A separate eight-configuration sweep varied local residuals and conditional mixing, which combines planes according to the input. More local freedom improved prediction loss, but the extra storage cost outweighed that improvement: the storage multiplier fell from 1.000x to 0.942x. Better predictions alone therefore do not establish better capability per byte.

Pretrained conversion

The representation was also tested by compressing SmolLM2-135M, an existing pretrained model. Each method received the same fitting and recovery process: adapting the compressed weights, then training to recover lost prediction quality. Compression ratios below are relative to the INT4 baseline. Loss is reported in bits per UTF-8 byte of text, so the denominator here is text size, not model storage. Lower loss is better.

Dimensional sharing compressed this model further than the tested lower-bit quantization methods. But structured pruning, which removes groups of model parameters, achieved essentially the same compression with slightly lower loss and a simpler construction. The higher compression ratio therefore does not establish an advantage specific to dimensional sharing.

What the experiments support.

The supported findings concern specific tasks, storage accounting and the recovery process. They remain useful, but none reverses the negative result for language capability per deployed byte.

Reusable engineering results

• Weight-space reconstruction is preferable to expanding activations for this implementation.
• Measure the full training step, because timing an isolated GPU operation does not establish an improvement in the complete system.
• CUDA Graph capture, fused AdamW updates and selected Triton epilogues improved the research implementation by reducing execution overhead and combining operations.
• Correctness checks on computation and stored model data caught defects that changed scientific conclusions.

The measurements constrain possible explanations.

Keep failed hypotheses and withdrawn claims visible.

This register records what was tested and why each claim was retained, falsified or withdrawn. A new proposal needs to explain what changes and why that change predicts a different outcome. Learning from a failed hypothesis does not make the hypothesis successful. Invalid or quarantined evidence cannot support a mechanism claim, and open claims remain unproven.

How the comparisons became more demanding.

Next tests need a reason to predict a different outcome.

The project remains active. Some tests examine unresolved choices within the current construction; others investigate different mechanisms or types of stored state. These are proposed tests and research directions, with no demonstrated advantage yet.

Where the evidence stops.

• From-scratch language training was measured at 20M and 50M only.
• SmolLM2-135M was used for pretrained compression, not an equivalent from-scratch campaign.
• Language capability is measured by prediction loss on held-out FineWeb-Edu text. Performance on downstream tasks has not been established by this measure.
• The controlled transformer changes the FFN rather than every memory-bearing component.
• The positive small-task composition result remains real even though it did not generalise.
• Varying the number of planes and testing neuron assignments under matched recovery remain open directions.
• Adaptive reconstruction and different memory objects lie outside the present falsification.
• Large-model storage and decode gains remain structural or projected, not measured deployment results.