Overview
Chinese Hamster Ovary cells manufacture roughly 70% of all recombinant pharmaceuticals — insulin, monoclonal antibodies, clotting factors. When you want a CHO cell to make a protein, you hand it a gene, and here there is a free parameter most people never touch: the genetic code is degenerate. 61 codons encode only 20 amino acids, so most amino acids have several synonymous codons. Swapping between them leaves the protein completely unchanged, but changes how fast and how reliably the ribosome can build it — which changes yield.
Choosing those codons well is called codon optimization, and it's usually done with hand-built heuristics like "use the most frequent codon for this organism." We built two transformers to do it instead. CHOExp learns the mapping from a DNA sequence to how strongly CHO cells actually express it. CHOFormer takes a protein sequence and generates a coding sequence for it, trained only on the genes CHOExp says are highly expressed. The result: optimized sequences reach a mean Codon Adaptation Index of 0.847 versus 0.654 for the natural sequences, and a Translational Adaptation Index of 0.682 versus 0.373. It's deployed as a web tool at choformer.com — paste a protein, get a gene.
Context
Every biologic that reaches a patient was expressed in a cell line first, and for most of them that cell line is CHO. Expression yield is therefore a direct input to how much a drug costs and how quickly it can be scaled — and it's determined partly by the gene sequence you hand the cell, before any bioreactor engineering happens.
The lever is synonymous codon choice. Because the code is degenerate, there are enormous numbers of DNA sequences encoding the identical protein, and they are not equivalent to the cell: tRNA abundances differ per codon, so some codons are read quickly and accurately while rare ones stall the ribosome, and the choice also affects mRNA folding and stability. The conventional approach is to substitute in each amino acid's most frequent codon in the host genome. That's a reasonable first-order rule, but it's context-free — it treats each position independently, ignores the surrounding sequence, and optimizes for a frequency table rather than for expression itself.
That gap is what makes this a learning problem. Sequence context matters, the objective we actually care about is measurable, and there is enough CHO transcriptomic data to learn from — so a model can be trained against observed expression rather than against a proxy heuristic.
Question
Can a sequence model learn CHO codon preference directly from expression data — and can a second model then generate, for an arbitrary protein, a coding sequence that looks like the genes CHO cells express best?
Method
Two models, and the link between them. The pipeline's key structural idea is that the two models compose: CHOExp learns what high expression looks like, and its judgments are then used to decide which genes CHOFormer is allowed to learn from. Rather than training the generator on all CHO genes — most of which are expressed at unremarkable levels — it's trained on the subset CHOExp scores highly, so what it imitates is the codon usage of genes that actually work.
CHOExp — predicting expression from sequence. Starting from 26,795 CHO genes with matched RNA expression values, genes with zero expression were dropped and the top 66% falling within three standard deviations were retained, leaving 13,253 genes. Expression was log-scaled and normalized to , then split 80/10/10. The model is an encoder-only transformer — 384-dimensional, 8 layers, 4 attention heads — reading DNA as 3-mer tokens, truncated or padded to 1024 tokens (3,072 base pairs). A classifier token is prepended, and its final hidden state passes through a linear head and a sigmoid to produce a normalized expression prediction. Trained for 10 epochs with validation after each.
CHOFormer — generating optimized codons. The training corpus began as ~97,000 CHO protein-coding sequences from NCBI, filtered to those between 300 and 8,000 base pairs (86,632 remaining), then deduplicated with cd-hit-est using an 8-word window at 90% nucleotide identity to give 47,713 non-redundant sequences. Clustering matters more than it sounds: without it, large families of near-identical paralogs dominate the gradient and the model overfits to whichever gene families happen to be numerous rather than learning general codon preference. Sequences were translated to amino acids with unnatural residues removed, then split 80/10/10.
The generator doesn't read raw amino acid letters. Protein sequences are embedded with ESM-2-150M, a protein language model whose representations already encode structural and evolutionary context, and those embeddings are projected into a higher-dimensional space before entering the decoder — two decoder layers with four attention heads, run causally so each codon is generated conditioned on the ones before it. The output logits are mapped to a probability distribution over a custom 3-mer tokenizer built so that every codon is exactly one token. That detail is load-bearing: with a generic nucleotide tokenizer the model would have to learn the reading frame, whereas a codon-level vocabulary makes synonymous substitution a single-token choice and the frame impossible to break.
Evaluation. The two standard measures both quantify how well a sequence's codons match what the host is equipped to translate. The Codon Adaptation Index is the geometric mean of per-codon relative adaptiveness across the gene,
where compares codon 's usage frequency to the most-used synonymous codon for that amino acid — so a gene made entirely of a host's preferred codons scores 1. The Translational Adaptation Index instead weights each codon by tRNA gene copy number, making it a more direct readout of how quickly the ribosome can actually service the sequence. Both were computed for CHOFormer's outputs and the corresponding natural sequences.
Result
Results
0.847
Mean CAI, optimized (from 0.654)
0.682
Mean TAI, optimized (from 0.373)
47,713
Non-redundant CHO genes
~70%
Of biologics made in CHO
Codon Adaptation Index rose from a mean of 0.6541 (± 0.0526) for the natural sequences to 0.8471 (± 0.0874) for CHOFormer's — roughly a 30% increase. Translational Adaptation Index rose from 0.373 (± 0.112) to 0.682 (± 0.209), an ~83% increase, and the larger relative gain on TAI is the more interesting of the two: TAI is weighted by tRNA availability, so improving it means the model is not merely matching a frequency table but selecting codons the cell is actually equipped to translate quickly.
The distributions tell a subtler story than the means. The optimized sequences sit in a tight band near the top of the range, while the originals cluster around 0.65 — but the optimized set also carries a long tail of low-CAI outliers, meaning the model is confident and near-optimal on most proteins and occasionally fails badly on some. Knowing which proteins fall in that tail is the obvious next question, and the honest answer is that we didn't get to it during the hackathon.
The larger caveat is worth stating plainly: CAI and TAI are proxies. They correlate with expression and they are what the field uses to benchmark codon optimizers, but neither is a wet-lab measurement of protein yield. Validating these sequences in actual CHO culture is the step that would convert this from a promising result into an established one.
Both curated datasets were published on Polaris Hub, and the whole thing was deployed as a public tool — a Next.js front end over a Flask API serving the PyTorch models on AWS — so that using it requires pasting in a protein sequence rather than reproducing a training run.
Reflection
The design choice I keep coming back to is using one model to curate the other's training data. It would have been easier to train the generator on all 47,713 genes and hope it averaged out to something good. Filtering through CHOExp instead meant the generator only ever saw genes the first model believed CHO cells express well — which turns "learn to write CHO genes" into the sharper objective "learn to write good CHO genes." That pattern, where a discriminative model shapes the data distribution a generative model learns from, seems broadly reusable whenever the quality label is easier to learn than the generation task.
The other lesson was about representation. Two decisions — embedding proteins with ESM-2 rather than one-hot amino acids, and tokenizing DNA as codons rather than nucleotides — did more work than any architectural choice. Both amount to handing the model a representation in which the thing it needs to learn is already easy to express: ESM-2 supplies structural and evolutionary context nobody would have hand-engineered, and the codon tokenizer makes synonymous substitution a single-token swap. The transformer on top is unremarkable, and it didn't need to be otherwise.