I Built My Own “Jev” and Fine-Tuned It on Banking Data

Models that do not write. They decide.

This tiny local model turns a banking message into a typed decision in about 35 milliseconds. No generated paragraph. No data-center GPU. I trained the whole thing in under 30 minutes on an Apple MacBook Air with 16 GB of RAM.

Everyone is talking about models that take text in and return typed choices with probabilities. It sounds like a new category of AI. So I built one.

Credit where it is due. This experiment was directly inspired by Duarte O.Carmo’s NobodyWho article, “Jev in 25 lines of Python.” That post showed how a local open model’s next-token logits can become a closed set of probabilities. I took that idea further by fine-tuning a small model on public banking data, measuring it on a held-out test set, and adding an explicit human-review threshold. Jev itself is TypeSafe AI’s product; this is an independent experiment, not a reproduction of its architecture.

To be precise, I built my own tiny, local, Jev-shaped decision model. It is not TypeSafe’s Jev and does not claim to reproduce its architecture. It takes the useful product idea seriously: stop asking a language model to compose an answer when the product needs a bounded decision.

I fine-tuned a 135M-parameter model on Banking77: 10,003 customer-support messages spanning 77 intents. The model sees a message such as:

Why was my card payment declined?

It does not generate a reply. It returns a probability distribution:

{
  "declined_card_payment": 0.9625,
  "reverted_card_payment": 0.0204,
  "declined_cash_withdrawal": 0.0092
}

Then deliberately boring policy code turns intent into action:

if confidence < 0.70:
    action = "human_review"
elif intent in security_intents:
    action = "secure_account_now"
elif intent in investigation_intents:
    action = "investigate_transaction"
elif intent in assisted_support_intents:
    action = "assisted_support"
else:
    action = "self_service"

That separation matters. The model estimates what the customer needs. Ordinary, inspectable code decides what the product should do about it.

Fine. But does my fake Jev actually work?

We evaluated on Banking77’s 3,080-message test split. The threshold curve below is descriptive test-set analysis; a production system must choose and lock its threshold on separate validation data.

Result	Before training	After training
Exact intent accuracy	1.62%	83.25%
Operational decision accuracy	47.99%	92.01%
Calibration error	27.39%	6.78%

The pre-training intent score is close to the 1-in-77 random baseline. The high pre-training action score is less impressive than it looks: many intents share the same broad action.

Probability lets the model decline to decide

With a 70% acceptance threshold, the model automatically routes 67.05% of messages. On that accepted subset, exact intents are 95.88% accurate and operational decisions are 98.45% accurate. The uncertain 32.95% go to a person.

Threshold	Automated	Decision accuracy
50%	83.38%	96.77%
70%	67.05%	98.45%
90%	41.56%	99.69%

More automation is available if we accept more errors. Less automation buys greater precision. The threshold makes that tradeoff explicit instead of hiding it behind a generated paragraph.

Is it fast?

Very. On the same 16 GB Apple MacBook Air used for training, across 1,000 warm single-message requests:

p50        35.16 ms
p95        47.43 ms
p99        57.88 ms
throughput 26.26 requests/second

Cold model loading took 1.73 seconds. The warm figures include tokenization, inference, probability normalization, thresholding, and action routing.

The full one-epoch fine-tune on all 10,003 training examples took less than 30 minutes on that laptop. That combination—a sub-30-minute training run and answers in tens of milliseconds—is what made the experiment feel less like a demo and more like a practical local component.

Run it yourself

This repository contains the exact implementation, the trained LoRA adapter, and the evaluation evidence behind every number above. Clone it, install the pinned dependencies, and make a prediction:

git clone https://github.com/shmcsensei/easy-jev-fine-tune.git
cd easy-jev-fine-tune
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

python banking77.py predict \
  "Why was my card payment declined?"

To recreate the adapter from the official training split and evaluate it once on the test split:

python banking77.py train \
  --examples-per-class 999999 --epochs 1 \
  --output adapter-banking77-reproduced

python banking77.py evaluate \
  --adapter adapter-banking77-reproduced \
  --threshold 0.70

The code is part of the post. The README explains the fast smoke-test path, full reproduction commands, evaluation, benchmarking, expected downloads, and sources of result variation.

There. I built my own “Jev.”

Well, Jev-shaped.

It is not the actual Jev, and it is not a production banking system. The probabilities are better calibrated after training, but they are not guarantees. Banking77 labels support intents, not fraud outcomes. The action mapping is prototype business policy and would need operational, risk, and compliance review.

The controversial bit is not that this replaces Jev. It does not. It is that a surprisingly large slice of the product experience is reproducible with a tiny open model, one classification head, some LoRA weights, and an honest threshold.

The useful part is real: a small local model can turn unstructured customer language into fast probabilistic decisions, expose uncertainty, and hand the hard cases to humans.