Hot off the Wires 🔥

We're eavesdropping on the smartest minds in research. 🤫 Don't miss out on what they're cooking up! In this section, we dissect some of the juiciest tech research that holds the key to what's next in tech.⚡

Recently, Trouble had an idea of integrating an AI agent powered by GPT-4 for The Vision Debugged to engage our readers with deeper conversations & answer their questions related to our newsletter content.

Sounds impressive, right? Well, not exactly for our wallets. Yeah, we referred to some quick analysis to realize that for a small business, using GPT-4 for customer support could translate to a hefty bill of over $21,000 a month. Yikes!

This is exactly where techniques like LLMingua come in as a lifesaver. Authored by researchers at Microsoft, the LLMLingua paper proposes a clever method to significantly reduce the cost of using black-box LLMs like ChatGPT & Claude through their APIs, without sacrificing their capabilities by shrinking the size of the prompts you feed them. Think of it like putting your LLM on a diet – it gets leaner and meaner, without sacrificing its brainpower.

Before we dive into LLMLingua's magic, let's address a couple of key concepts.

LLMLingua leverages perplexity to compress prompts fed to LLMs. The core idea is to remove unnecessary information from the prompt while ensuring the LLM can still understand the context and answer your questions accurately. The important idea to note here is that “removing tokens with lower perplexity has a relatively minor impact on the LLM’s comprehension of the context”

Under the Hood

Provided with the actual prompt & a target compression ratio (or target token number), LLMLingua employs a two-pronged approach using a locally run LLM for prompt compression: coarse-grained and fine-grained compression.

Let’s break them down step by step.

Coarse-Grained Compression through Demonstration-Level Perplexity

Typical chain-of-thought (CoT) prompts, or those involving retrieval augmented generation (RAG) are long & include 3 key components:

• Instruction: This set the background for the LLM about what to do & how to do it

• Demonstrations: These are examples/context documents that guide the LLM (CoT examples or retrieved snippets)

• Question: This is what you want the LLM to answer

While the instruction & question are critical for the LLM response, demonstrations often tend to be redundant (indicated by the “perplexity” of the demonstrations). LLMLingua smartly picks up on this & uses a Budget Controller to strategically allocate compression budgets for the various components such that demonstrations with lower perplexity likely (containing less crucial information) can be compressed more aggressively.

The outputs include a target prompt containing the instruction & question, along with a reduced number of demonstrations, and the adjusted compression rates for each of the prompt components.

Fine-Grained Compression through Token-Level Perplexity

The remaining tokens across the target prompt may also have tokens containing redundant information. So now, LLMLingua focuses on this text, & uses a technique called Iterative Token-level Prompt Compression (ITPC). This leverages "token-level perplexity" to pinpoint individual words or phrases that can be removed without significantly impacting the meaning.

Basically, LLMLingua breaks the target prompt from the budget controller into bite-sized chunks, and analyzes each token in each chunk iteratively, based on how well the LLM understands the context so far (by computing the perplexity of the token). For each segment, a threshold perplexity is chosen based on the conditional probability of the segment and the adjusted compression rates. Only the tokens from the segment that pass the threshold are retained in the final compressed prompt.

For thoroughness, when working on a chunk, token-level perplexities are computed based on the compressed previous chunks & the current chunk under compression.

Real-World Impact: Making LLMs More Affordable

The results speak for themselves. LLMLingua can achieve compression ratios of 3x to 20x, meaning the compressed prompt can be up to 20 times shorter than the original! This translates to massive cost savings without sacrificing accuracy.

Evaluation metrics on various datasets show that black box LLMs using compressed prompts performed just as well, or slightly better, than those using the full-length versions.

Techniques like LLMLingua open doors for the wider adoption of LLMs across various industries. It is useful in scenarios where LLMs are used with very long prompts, such as chain-of-thought (CoT) prompting and in-context learning (ICL). Examples of such applications could be question-answering systems or creative text-generation tasks that benefit from extensive context.

Beyond LLMLingua

LLMLingua paves the way for a more advanced technique called LongLLMLingua. This addresses the challenge of extremely long prompts (particularly in RAG applications) and a phenomenon known as the "Lost-in-the-Middle" problem.

LongLLMLingua tackles this by incorporating "question awareness" into its compression strategies. Intuitively, it modifies the Budget Controller to select documents based on their “importance” to the question (computed perplexity of the question, if it is concatenated after the document) and uses a “contrastive perplexity” in ITPC to distinguish tokens relevant to the question & improve key information density. For more details, interested folks can check out the LongLLMLingua paper.

How You Too Can Try Out LLMLingua

Here's the cool part: you can try it out yourself, right away! Dive into the code over at the LLMLingua GitHub repository or play around with a live demo on Hugging Face 🤗.

Developers, you're in luck too! LLMLingua is already integrated into libraries like LlamaIndex and LangChain (two widely used RAG frameworks), making it super easy to use in your projects.

Spark and Trouble are keeping a close eye on these developments – after all, who wouldn't want a more affordable and efficient way to chat with a large language model?

10x Your Workflow with AI 📈

Work smarter, not harder! In this section, you’ll find prompt templates 📜 & bleeding-edge AI tools ⚙️ to free up your time.

Fresh Prompt Alert!🚨

Feeling overwhelmed by that ever-growing content calendar?
Wish you had a magic social media schedule that practically posts itself?

We got you! This prompt will be your social samurai, crafting a month-long plan for your business. Just fill in the blanks, post & watch your engagement soar! 💯

3 AI Tools You JUST Can't Miss 🤩

• ✍️ Becca - Find the latest trends in your niche to create engaging posts that sound just like you

• 🔮 Smartli - Generate SEO-friendly and high-quality product descriptions 10x faster

• 📽️ Glato - Create short video ads that sell with AI creators.

Spark 'n' Trouble Shenanigans 😜

Stuck on logo ideas? This image prompt template sparks instant logo & icon inspiration for your brand or app, in any style! Try it out on Midjourney or click here to test it on Microsoft Designer.

simple [style] icon for an [describe the app] app, [primary color] colour with [background color] background, ui/ux, ui --s 75

Check out some of our awesome creations using this image prompt template 👇