Measured on real embedding models โ€” RTX 5090 validation run

Your embeddings can't tell ๐Ÿ”ฅ apart from ๐ŸŽ‰ or ไนฐ apart from ๅ– โ€” and that's silently breaking your search.

ไนฐ buy and ๅ– sell โ€” opposite meanings, Chinese, Japanese, Korean (CJK) script โ€” embed to cosine 1.0000 on MiniLM. Not similar: the identical vector. No ranker, threshold or reindex can separate them, because there is nothing left to separate. Through the gateway that pair drops to 0.9015 โ€” distinguishable at last.

Popular embedding models collapse emoji and CJK text into near-identical vectors. 96.25% of distinct emoji collapsed into duplicates on MiniLM. 89.6% of compound emoji sequences collapsed on a Chinese-optimized model. Semantic Integrity Gateway is a drop-in middleware that fixes it before your embedder ever sees the input โ€” no retraining, no model swap.

Get an API key Run the 10-second diagnostic See the collapse data โ†“
RUN THE 10-SECOND DIAGNOSTIC Live API ยท no key needed
See what your embedding pipeline silently does to this text.
Emoji: 96.25% โ†’ 0% collapse ยท CJK: 3/3 identical-vector pairs broken ยท p99 152ms @ 1000 concurrent
Built for teams running
RAG pipelines Semantic search Content moderation Dedup & clustering Bilingual support tooling
01 โ€” The Bug

Your embedding model is lying about similarity.

We ran independent neural validation on real embedding models (RTX 5090, full precision, no shortcuts). The result: emoji and CJK text routinely collapse into duplicate or near-duplicate vectors. If your product does search, retrieval, moderation, or dedup on this content, it is already misclassifying it โ€” silently.

96.25%
of distinct emoji collapsed into duplicate vectors
MiniLM โ€” only 18 of 480 test emoji produced genuinely distinct embeddings.
89.6%
of compound / ZWJ emoji sequences collapsed
bge-base-zh-v1.5 (Chinese-optimized) โ€” plus 86.5% of simple emoji collapsed too.
50%
of simple emoji still collapsed on a stronger model
bge-m3 handled compound/ZWJ sequences perfectly (240/240 distinct) โ€” but simple emoji still fail at a coin-flip rate.

Why this matters: if two visually and semantically different emoji โ€” or two different Chinese phrases โ€” produce the same or near-identical vector, your vector database can't tell them apart. Search returns the wrong result. Your RAG pipeline retrieves the wrong context. Your moderation model can't distinguish ๐Ÿ”ฅ (hype) from ๐ŸŽ‰ (celebration) from ๐Ÿšจ (alert). This isn't a hypothetical edge case โ€” it's a measured, repeatable failure mode in models you are probably already running in production.

Read the research on the underlying tokenizer/embedding collapse โ†’
02 โ€” The Fix

A gateway, not a model swap.

Semantic Integrity Gateway sits between your application and your embedding call. It applies a hardened semantic disambiguation layer to emoji, CJK, and bilingual text before the vector ever gets computed โ€” preserving distinct meaning instead of losing it to collapse. No retraining. No new embedding model. No changes to your vector database.

1

Ingest

Your text hits the gateway exactly as it does today โ€” same request shape, drop-in FastAPI layer in front of your existing embedding endpoint.

2

Disambiguate

Emoji, CJK spans, and bilingual mixes are detected and passed through a versioned semantic ruleset that resolves whole-sequence meaning โ€” not code-point spelling.

3

Preserve original

The native input is never discarded. A provenance-bearing companion representation travels alongside it, restorable byte-for-byte.

4

Embed & ship

Your existing embedding model receives disambiguated input and produces vectors that actually separate distinct meaning โ€” search, RAG, and moderation start working correctly.

before_after.diff 
# before โ€” direct call, silent collapse risk
- vector = embed_model.encode(user_text)

# after โ€” one line, same interface
+ vector = sig_gateway.encode(user_text)
  # emoji + CJK + bilingual spans disambiguated first
  # original preserved, audit trail attached, then embedded
03 โ€” Validation

Not a research toy. Production numbers.

We hold the gateway to the same bar as the models it protects: held-out accuracy tests and real concurrent load, not cherry-picked demos.

โœ“
100%
accuracy on unambiguous cases
40-item bilingual (English/Chinese mixed) holdout test.
โœ“
90%
accuracy on ambiguous, double-meaning cases
The hard cases โ€” where a phrase or emoji genuinely carries two plausible meanings.
โšก
152ms
p99 latency at 1,000 concurrent requests
0 errors under sustained concurrent load โ€” production-grade, not a notebook demo.
Emoji embedding collapse by model โ€” before Semantic Integrity Gateway
Model Simple emoji collapse Compound / ZWJ collapse Notes
MiniLM 96.25% 96.25% Only 18/480 test emoji produced distinct embeddings.
bge-base-zh-v1.5 86.5% 89.6% Chinese-optimized model โ€” still fails badly on emoji.
bge-m3 50% 0% (240/240 distinct) Best of the three, but a coin-flip on simple emoji.

CJK opposites โ€” measured cosine, before and after the gateway

Eight semantic-opposite pairs, embedded with all-MiniLM-L6-v2 at full precision. "Before" is the raw pair. "After" is the same pair passed through /v1/normalize. Reproduce any row with a free key in one call.

CJK semantic-opposite pairs โ€” cosine similarity, MiniLM (lower = more distinguishable)
Pair Meaning Before After Status
ไนฐ / ๅ– buy / sell 1.0000 0.9015 Total collapse broken
็›ˆๅˆฉ / ไบๆŸ profit / loss 1.0000 0.7739 Total collapse broken
ๆ‰นๅ‡† / ้ฉณๅ›ž approve / reject 1.0000 0.8548 Total collapse broken
ไธŠๆถจ / ไธ‹่ทŒ rise / fall 0.6256 0.6685 Already distinct โ€” gloss added
็™ปๅฝ• / ้€€ๅ‡บ log in / log out 0.6465 0.7088 Already distinct โ€” gloss added
ไนฐๅ…ฅ / ๅ–ๅ‡บ buy in / sell out 0.6465 0.6961 Already distinct โ€” gloss added
ๅŒๆ„ / ๆ‹’็ป agree / refuse 0.5404 0.5961 Already distinct โ€” gloss added
ๅšๅคš / ๅš็ฉบ go long / go short 0.4676 0.6884 Already distinct โ€” gloss added
Pairs at cosine โ‰ฅ 0.95 (unrecoverable) 3 / 8 0 / 8 Every total collapse eliminated

What we do not claim. Mean cosine across all eight pairs is essentially unchanged (0.7408 โ†’ 0.7360). The gateway is not a general-purpose "push everything apart" trick, and pairs already separated can move slightly closer because the appended English gloss shares surface tokens. The specific, load-bearing claim is narrower and stronger: where a pair was mathematically indistinguishable โ€” cosine 1.0000, the identical vector โ€” it no longer is. That is the failure no amount of downstream ranking can fix. Characters whose sense is genuinely context-dependent (่กŒ hรกng/xรญng) are deliberately left unglossed and flagged requires_multilingual_route rather than guessed at.

04 โ€” Pricing

Priced like infrastructure, not like a feature.

Metered by characters processed through the gateway. No seat licenses, no per-request tax on top of your embedding bill โ€” you pay for disambiguation, not for volume you'd be paying anyway.

Prices below are BTC/on-chain prices โ€” our base price. Paying by credit card or fiat costs a flat 5% more (fiat price = BTC price ร— 1.05). Pay on-chain or over Lightning and save 5%.

This toggle only changes the prices shown. To actually pay, hit Buy now below โ€” you'll get a Bitcoin QR code and invoice, or card options.

Developer

$0/mo

For evaluation, prototyping, and low-volume production traffic.

  • Up to 10M characters / month
  • Full disambiguation ruleset (emoji + CJK + bilingual)
  • Sidecar & replace modes
  • Community support
Start free
Most common

Growth

$300/mo

Flat monthly rate with a 1 billion character allowance โ€” then just $0.30 per additional 1M characters.

  • 1B characters / month included
  • Pennies per additional 1M characters
  • Priority routing & SLA-backed p99 latency
  • Audit trail + exact-restore guarantee
  • Email & Slack-connect support
Buy now โ€” BTC or card

Enterprise

$8,000/mo starting

Flat-rate contracts at scale, VPC / on-prem deployment, dedicated support.

  • Volume-committed flat monthly fee
  • On-prem / VPC deployment option
  • Model-version canary + migration tooling
  • Dedicated solutions engineer
Talk to sales

Real customer economics โ€” 20B characters/month example

Computed directly from the published Growth-tier rates above โ€” not rounded marketing math.

Monthly volume
20B characters
Baseline monthly spend
$400,000 at $20/M chars
Fraction of failures avoided
10% conservative estimate
Gross cost avoided
$40,000 / month
Gateway monthly fee
$300 Growth tier, 1B chars included
Gateway overage cost
$5,700 19,000 ร— $0.30 per extra 1M
Net monthly savings
$34,000 / month
Customer ROI on gateway spend
6.7ร— return
Break-even requires avoiding just 1.5% of failure-driven cost โ€” the measured collapse rates above are an order of magnitude larger than that threshold.
05 โ€” Details

Questions engineers actually ask.

Does this replace my embedding model?

No. It runs in front of it. You keep your existing model, vector database, and retrieval stack โ€” the gateway just makes sure distinct emoji/CJK/bilingual input actually produces distinct vectors before it hits your embedder.

Do you ever discard or rewrite my original text?

Never. The native input is preserved byte-for-byte with a versioned, provenance-bearing companion representation traveling alongside it. Restoration is exact in 100% of tested cases.

What happens with emoji or characters you don't recognize?

Unknown sequences are preserved untouched and flagged rather than guessed at โ€” a deliberate fail-safe for Unicode and model-version drift.

How was this validated?

Neural validation was run on real embedding models on an RTX 5090: MiniLM, bge-base-zh-v1.5, and bge-m3. Emoji fix accuracy was measured on a 40-item bilingual holdout set, and throughput was load-tested at 1,000 concurrent requests. CJK disambiguation was measured separately on eight semantic-opposite pairs with all-MiniLM-L6-v2 at full precision: three pairs sat at cosine 1.0000 (literally the same vector) and all three were broken, with zero pairs remaining above 0.95. We publish the pairs that barely moved alongside the ones that did โ€” see the CJK table above and the "what we do not claim" note under it. Read the research for full methodology.

Does the CJK fix cover every Chinese character?

No, and that is deliberate. The built-in lexicon covers 130 high-confidence senses concentrated on the vocabulary where collapse does commercial damage โ€” trading direction, money flow, account actions, and approve/reject decisions. Characters whose meaning genuinely depends on context (่กŒ hรกng "row/firm" vs xรญng "walk/okay") are intentionally not guessed. Those spans come back untouched with requires_multilingual_route: true so you can route them to a multilingual embedder. A wrong gloss would move the vector somewhere confidently wrong, which is worse than an honest flag. You can override or extend any entry per-request with chinese_glosses, and a caller-supplied gloss always wins over ours.

Is this a translation service?

No. We don't translate or replace Chinese, Japanese, or Korean text with English. We preserve the native original and add a disambiguating semantic companion โ€” native-language retrieval quality is never sacrificed.

What's the integration effort?

A drop-in FastAPI-compatible endpoint. Most teams swap one function call โ€” embed_model.encode() โ†’ sig_gateway.encode() โ€” and ship the same day.

Stop shipping embeddings that can't tell your data apart.

Get an API key, run your own emoji/CJK collapse test, and see the difference in an afternoon.

Read the research

Developer tier is free โ€” no credit card, no payment. Paid tiers: pay by Bitcoin for instant automatic access (base price), or by card for 5% more. Cancel anytime.