- The Vision, Debugged;
- Posts
- NeuroBind can Visualize Thoughts: Is Sci-Fi becoming Reality?
NeuroBind can Visualize Thoughts: Is Sci-Fi becoming Reality?
PLUS: The AI Fever is on at the Paris Olympics too
Tezan Sahu & Sandra Anil
August 6th, 2024
Howdy fellas!
Remember the excitement of fitting that last puzzle piece? Spark and Trouble are back, piecing together a mind-bending edition that bridges brains and bytes. From neural signals to AI-powered productivity and cold email mastery, we're about to unravel tech's latest enigmas. Ready to snap it all into place?
Gif by ExplainerStudio on Giphy
Hereās a sneak peek into todayās edition š
š§ Discover how NeuroBind takes us a step closer to mind-reading
š© Craft & test your email campaigns with this kickass prompt
šŖ 3 AI Tools to Supercharge Your Productivity
š Check out how the custom GPT for the 2024 Paris Olympics fares
Time to jump in!š
PS: Got thoughts on our content? Share 'em through a quick survey at the end of every edition It helps us see how our product labs, insights & resources are landing, so we can make them even better.
Hot off the Wires š„
We're eavesdropping on the smartest minds in research. š¤« Don't miss out on what they're cooking up! In this section, we dissect some of the juiciest tech research that holds the key to what's next in tech.ā”
Remember that mind-bending scene in "Inception" where the characters dive into dreams, reconstructing entire worlds from memories? Or perhaps you've marveled at the neural interfaces in "The Matrix" that translate brain signals into a virtual reality.
These sci-fi concepts have long captivated the imaginations of our duo - Spark & Trouble, but what if we told you that we're inching closer to making them a reality?
Scientists have long grappled with understanding the complex language of the brain. Neural signals, captured through various methods like EEG, fMRI, and calcium imaging, offer a glimpse into the mind's workings. Of late, several interesting studies have proposed innovative ways to reconstruct images accurately from brain activity.
Amidst this landscape, enter NeuroBind - a groundbreaking AI model that unifies various neural signals into a single, powerful representation, transforming how we understand and interact with brain activity.
Forging the Fundamentals
As always, before we dive into the nitty-gritty, let's break down some brain-twisting jargon:
Neural Modalities: These are different types of brain signals recorded from different devices, such as EEG (electroencephalography), fMRI (Functional Magnetic Resonance Imaging), Calcium Imaging, and Spiking Data. Each modality provides unique insights into brain activity.
Visual Stimulus: This is an image that is shown to a subject while recording their brain activity, while they try to think about it
Embeddings: Imagine turning complex data into a mapping where similar items are closer together. That's what embeddings doāthey convert data like words or images into numerical representations.
Contrastive Learning: This technique teaches a model to keep similar items close and dissimilar ones apart in its embedding space. This can be achieved by training models to minimize various interesting types of loss functions. Check out this detailed article to understand them.
Zero-Shot Methods: In these methods, AI models perform tasks without having seen any examples of those tasks during training. Basically, a āshotā is an example of how the task is to be performed. It's like answering a question about a topic you've never studied, using general knowledge and context clues.
Linear Probing: This evaluates a pretrained model by freezing the model weights and adding (& training) a simple linear-layer classifier on top, and then checking its performance. It's a way to see if the model has learned useful features.
So, whatās new?
Traditionally, processing neural data has been hindered by the vast differences between various signal types. Different signal types (or modalities) capture distinct aspects of brain activity, making it tricky to create a unified understanding. Moreover, the scarcity of large-scale, high-quality neural datasets has posed a significant challenge in training high-quality models.
Despite these hurdles, researchers have made fascinating strides in reconstructing stimulus images from neural activity using GANs and diffusion models. Recent studies like MindEye2 have even managed to reconstruct images with high accuracy from fMRI data with minimal training, by combining retrieval techniques with deep neural networks. However, these approaches often focus on a single neural modality.
Reconstruction of image stimuli using fMRI recordings (source: MindEye2 paper)
NeuroBind takes a giant leap forward by studying different neural modalities and utilizing them effectively for various neuroscience tasks. The hypothesis is simple yet profound: a single, well-learned embedding, aligned with certain pre-trained image embeddings, could represent different neural modalities, allowing us to study brain signals as interconnected phenomena.
Under the hoodā¦
Whatās the secret sauce behind NeuroBindās revolutionary approach?
Well, as the name suggests, NeuroBind ābindsā brain signals with external modalities like vision and language.
Now what does that even mean?
In simple language, it means encoding the various neural modalities into a unified representation (embedding space) that closely aligns with some existing embedding space trained for vision & language modalities (such as OpenAIās CLIP - Contrastive Language-Image Pre-Training).
And why is it needed?
Neural data is scarce, making it challenging to train effective models directly on this data. However, we have sophisticated models trained on images and text. By aligning brain signals with existing image embedding spaces, we can leverage these off-the-shelf models for diverse neuroscience research.
How does it work?
Learning Multimodal Representations of Brain Activity through Pre-Trained Image-Language Embeddings (source: NeuroBind paper)
Dataset Creation: Subjects (both human and, yes, even monkeys for some neural activity recording) view images (visual stimulus) while their brain activity is recorded.
Embedding Alignment:
NeuroBind uses a frozen CLIP image encoder to create visual embeddings for the visual stimuli
For each neural modality, a separate ātrainableā encoder with a Vision Transformer (ViT) backbone was set up to generate the neural embeddings
Contrastive learning is used, by computing InfoNCE loss between the visual & neural embeddings - this pulls the corresponding neural-visual pair close together and pushes them away from the other pairs
How does this matter?
Once this binding is complete, NeuroBind embeddings can be used for various intriguing tasks:
Retrieving Neural Signals: Finding neural signals of different modalities that represent the same or similar image modalities.
Semantic Classification: Classifying neural signals based on the semantic category of their visual stimulus. NeuroBind's zero-shot classification outperforms existing baselines, and linear probing enhances this performance further.
Linear probing also validated that different types of brain signals are complementary, and using them in a combined way (concatenating) can improve downstream task results
Image Reconstruction: This is, by far, one of the most interesting use cases - generating images from brain signals, allowing us to visualize thoughts. NeuroBind's use of off-the-shelf image synthesis models generates semantically consistent images in a zero-shot manner.
Zero-Shot Image Reconstruction using NeuroBind embeddings with a diffusion model (source: NeuroBind paper)
NeuroLLM: Combining NeuroBind encoders with a large language model (like LLaMA-7B) enables the interpretation of neural signals and the ability to describe them in words, providing insights that are otherwise non-trivial for humans - imagine being able to actually āread someoneās mindā. How cool is that!?
Using NeuroBind embeddings with an LLM to create NeuroLLM (source: NeuroBind paper)
NeuroLLM in action, interpreting & explaining the visual stimulus from the brain activity (source: NeuroBind paper)
Honestly, this isn't just cool science ā NeuroBind could truly revolutionize multiple fields:
Healthcare: Improved brain-computer interfaces for paralyzed patients or early detection of neurological disorders.
Education: Personalized learning experiences based on how individual brains process information.
Entertainment: Next-level video games or VR experiences controlled by thought alone.
Marketing: Understand consumer reactions at a deeper level (though this raises some ethical questions!).
Accessibility: New ways for non-verbal individuals to communicate.
As we continue to unravel the mysteries of the brain, technologies like NeuroBind are paving the way for a future where our thoughts and perceptions can be translated into a universal language of bits and pixels.
Who knows? Maybe one day soon, Spark will be able to snap a mental picture of the shenanigans that Trouble is up to. Until then, keep dreaming big ā your brain is already speaking volumes, and we're getting better at listening every day.
10x Your Workflow with AI š
Work smarter, not harder! In this section, youāll find prompt templates š & bleeding-edge AI tools āļø to free up your time.
Fresh Prompt Alert!šØ
You know that awkward feeling when you send a cold email, and it feels like shouting into the void?
Spark & Trouble have been there too. That's why this week's Fresh Prompt Alert is your ticket to becoming a cold email maestro! š
Whether you're a smooth-talking sales ninja or a shy dev stepping into biz-dev shoes, this prompt's got your back. It'll help you whip up not one, but two email variations faster than you can say "A/B testing." š
Ready to turn your inbox into a deal-closing machine? Give it a shot š
Act as a seasoned cold email writer. Craft a Compelling Cold Email to [Prospective Clientās Role] of [COMPANY NAME]. As a [Your Profession], my goal is to engage [Prospective Client Name], who operates in [Type of Business]. The focus is on a captivating subject line that sparks curiosity and eagerness to open the email.
Craft a compelling subject line that captivates my client and prompts them to open the email, emphasizing [Main Topic/Benefit]. Write this subject line keeping in mind reader psychology & top copywriting hacks.
Personalize the introduction, congratulating the recipient about the amazing job they have done with their product/service, later pointing out the common pain point they maye be facing.
Introduce my [product or service] to [ideal customer persona], addressing the common pain point.
Persuade [ideal customer persona] to sign up for [product or service] by offering a special promotion or discount. Showcase the value proposition of [product or service] and provide a persuasive response to potential cost-related hesitations.
Include a relevant case study or success story demonstrating the effectiveness of my product/service.
Develop an engaging call to action that prompts a click, ensuring higher conversion rates by emphasizing the time-saving and efficiency benefits it offers for their teams
Optimize the email content for a friendly and conversational tone. Keep it casual and personalize the message.
Perform this entire exercise twice to generate 2 variations of this email suitable for A/B Testing.
3 AI Tools You JUST Can't Miss š¤©
Spark 'n' Trouble Shenanigans š
Olympic fever is in full swing, and we couldn't help but notice the AI-powered ads from Google and Salesforce popping up between events. Reminds us of Microsoft Copilot's Super Bowl debut, doesn't it? AI's making quite the splash in the sports world!
Speaking of AI and the Olympics, have you heard about the 2024 Paris Games GPT? It's been causing quite a buzz among AI and sports fans alike. This custom GPT is supposed to be your personal Olympics assistant, telling you when events are airing and even converting times to your local zone. Sounds pretty nifty, right?
Well, Spark and Trouble decided to put it to the test and let's just say... while it did do well for most of our questions, it did screw up occasionally. š¤
The GPT gave reasonably good results for a bunch of questions (source: captured by authors)
We caught this GPT on what seemed like an off day. The results were, shall we say, less than gold-medal worthy.
GPT totally missed out on the early red card, the heroics of Shreejesh to see India through in the shootouts during the quarter-finals, while also hallucinating the incorrect score (source: captured by authors)
If you ask us, you might be better off sticking with Copilot or Gemini for your Olympic updates. Sometimes the classics just can't be beat!
Have any of you given the Paris Games GPT a try?
We'd love to hear about your experiences. Did it score a perfect 10, or did it fumble the ball?
Drop us a line and let us know!
Well, thatās a wrap! Until then, |  |
 |  |
Reply