leftbyte

Using Langchain, Pinecone, and Arize for Troubleshooting Search and Retrieval

2023-07-19T00:00:00+00:00

I recently watched this video on troubleshooting search and retrieval that was ostensibly a tech marketing workshop from Arize on their Phoenix observability tool. The video and sample code was a good introduction to Langchain, Pinecone and one approach on how to use an LLM on a custom set of documents. Langchain is a fantastic library that provides the many components and tools needed to build an application powered by language models. Pinecone is a cloud-based vector database.

The workshop go through ingesting a custom document store and using the generated embeddings to augment a prompt to ChatGPT. The materials also show how to interpose on the interfaces to collect some useful trace data to feed to Phoenix, and then inspect that data to understand the performance of the search and retrieval application.

Overview

Phoenix provides a locally hosted app that displays a reduced dimension visual of the embedding space of the prompt (input question) and supporting documents. E.g. you can see a point cloud of the prompt along with the supporting documents. Using the UI, you can select from different metrics and views to drill down to why certain responses are incorrect.

Bad responses (hallucinations): select ‘Color By’: dimension, user_feedback = dimension, and sort by lowest metric. When you look at the cluster with the lowest metric, you see a cluster with all the feedback being thumbs down (this is feedback from real users). Looking into this cluster’s data, you see that all of the questions are related to cost, which doesn’t have any good supported document responses. I.e. there is no documentation with any pricing information in the retrieved pieces of context. The problem with this information is that there is no ground truth and you’re just going by user feedback.
KB coverage - Query Density: select “Euclidian Distance” as the metric. You can see from this view that there are some questions that are clustered and far away from the context (documents). This way of looking at the clustering is agnostic of external feedback and lets you get a sense of the coverage of the support context. The problem with this view is that it won’t help you pinpoint any fine grained problems and is a pretty broad view of your data.
View Similar documents using cosine similarity - View the documents that have a high cosine similarity to the prompt. Select context_similarity_0 and view those how the cosine similarity score related to the prompt. This method lets you drill down to the verbatim documents that are being used to respond to the prompt. You’re also able to see how this metric can provide incorrect results because there are many documents that have very similar words and phrases but are simply irrelevant. This view shows how similarity != relevance.
View Relevance - The workshop notebook shows how to take another step in the system and use the LLM to rate the relevance of the documents to the prompt. This seems to be another way to get “external” feedback when human feedback is difficult to get. This shows that gpt-3.5 is pretty good at labelling relevance, but is not 100% accurate and that we still need some ground truth.

Learnings

Overall I enjoyed this overview of how to write a chatbot that can search a custom set of documents. This example also shows the pitfalls of language models and how these systems don’t have an easy way to provide a ground truth. If the application can tolerate some inaccuracy then this method can be applied. However, if the results aren’t 100% accurate then the loss of user confidence and tarnishing of the brand far outweighs the cost of building a robust search and retrieval system for a knowledge base.

In the Q&A section, they referenced Travis Fisher’s tweet on how to improve your application. I looked that up and here’s his advice on how to use LLMs effectively.

Start simple - if results are lacking, trying breaking up into subproblems or gradually moving up the ladder of complexity.

Prompting (i.e. prompt engineering)
Few-shot prompting i.e. prompt engineering)
Retrieval + prompting (using tools like langchain, llamaindex, etc.)
Iterative refinement (using tools like langchain, llamaindex, etc.)
Fine-tuning a hosted model
Fine-tuning an OSS model
Training an OSS model from scratch
Building a custom model from scratch

Is it sweet or savory?

2023-07-13T00:00:00+00:00

Earlier this past winter I took the fastai course to better understand the state of the art of machine learning. The goal of the course is to bring more people into the field of machine learning and fast.ai’s library lowers the barrier to entry into the field. The first class really showed how easy it is to get good results using the tools that people in the field have built.

Deep learning is for everyone.

You don’t need to understand a lot of math.
You don’t need a lot of data. Sometimes <50 items is sufficient, depending on the context and application.
You don’t need lots of expensive computers.

Learnings

Underestimating the constraints and overestimating the capabilities of deep learning may lead to poor results
Overestimating the constraints and underestimating the capabilities of deep learning may lead prevent you from exploring solvable problems
Design a process through which you can find the specific capabilities and constraints related to your particular problem

High level process

Make sure you have some data before starting, but don’t wait for perfect or complete data.
Start with setting up an end-to-end pipeline.
Set up your pipeline for fast iterations. Do things like reducing your training data size, downsample images, etc.
Integrate and deploy early and often so that you can test your assumptions.

Homework

In the first class, we were asked to build a simple visual classifier by fine tuning a pre-trained classifier. Using the fast.ai library and readily available images, I was quickly able to create a classifier that was able to distinguish between sweet and savory foods from a picture. I then published that model to a hugging face space: sweetOrSavory

I trained this with about ~500 images of sweet and savory foods I collected from a web search using these search terms:

sweet_foods = ['cakes', 'cookies', 'ice cream', 'pies', 'puddings', 'candy', 'soda', 'dessert', 'fruit']
savory_foods = ['dinner food', 'lunch food', 'sandwiches', 'pizza', 'casseroles', 'chips', 'crisps', 'fast food', 'pasta meals']

I started with the resnet18 model with a 20:80 validation/training split. The fine tuning took about 5 minutes and resulted in ~95% accuracy. It was also very simple to export and load up the model into an app on HuggingFace. Overall, this first dip into fast.ai’s way of doing machine learning was very fun and rewarding.

Hillbilly Chat

2023-06-27T00:00:00+00:00

Joining the crowd of folks interested in the current advances in machine learning and specifically the LLVMs being produced by OpenAI, I thought I’d create a simple chatbot. I have always loved how people say certain things in regional areas (like the south in the United States), so I thought a translator to “hillbilly speak” would be fun.

Initial stab

I looked at some examples and cobbled together something in a few minutes that used OpenAI’s gpt-3.5-turbo model. The API is fairly straight forward and it’s interesting that all the state is saved on the client, but it seems like the future bottleneck will be the memory limit of the messages context. After configuring the OpenAI API key and dealing with the pricing plan and billing, I was able to share the demo quickly to my friends using gradio.

Publishing to the Internet

After seeing that my friends had so much fun with it, I then wanted to push this out to the Internet to let others enjoy the banter. My first thought was to review the security issues. There’s not much exposed with this app but I did worry about Internet-scale and how the app would run up my OpenAI API bill, so I implemented some rate limiting to throttle the requests rate.

The actual publishing to a hosting service using gradio and HuggingFace was very simple. Within the terminal I simply used ‘gradio deploy’ and that workflow walked me through pushing it to a HuggingFace space. I ran into two issues:

README.md Configuration error: The HuggingFace Space build failed and said that there was a README.md configuration error. With some quick Googling about the issue I saw that I needed some frontmatter as described in the HuggingFace Spaces Configuration reference.

---
title: hillbilly-chat
app_file: billy.py
sdk: gradio
sdk_version: 3.35.2
emoji: 🤠
colorFrom: orange
colorTo: brown
pinned: true
---

The build failed again due to “RuntimeError: Share is not supported when you are in Spaces”. Looking at my app, I saw that I had gr.Interface(...).launch(share=True), which I simply changed to false.

The app was then deployed and you can now play around with the hillbilly-chat!

To try it out, hit the link and enter a phrase into the input box and see what a hillbilly would say.

Talking about it

I’m in the process of taking Jeremy Howard’s excellent fast.ai course in which he espouses the value of talking about your work. This is why I’m blogging about this now. My last step in this little project was to create a blogging site and post to talk about my fun hillbilly-chat project. I don’t expect to get anybody reading this, but it’s useful to shed that fear of showing the world what you’ve done.

Next steps

I tried changing the prompts so that more regional dialects and colloquialisms would be returned but it seems like the only responses are from a “United States Southern” region, which starts to get stale quickly. My assumption is that the reason for this regionality is because the corpus of data ingested by GPT-3.5 is very US centric and doesn’t discern between regional differences. The ideas to try to fix this are to 1. create my own vector database with a documents from different regions (books, articles, cartoons, etc) and feed that as context prompts into ChatGPT or

try to fine tune a smaller language model with that same corpus of data. Seems like access to good data is at the core of the issue.