I’m looking for guidance on using AI/LLM tools and (API and standalone) libraries to analyze free-form responses from large surveys (10,000 or even 100,000 entries).
Specifically, I would like to learn:
Which API-based and standalone (non-API) libraries are best suited for this task?
How can I get the following information once I obtain a general summary and the list of the main topics:
The precise number of replies associated with each identified topic.
A list of the original observation indices within the dataset for every reply that mentions a given topic."
For these types of tasks I’ve been using LangChain to build agentic workflows, which has prebuilt modules for API interactions with hundreds of cloud and local AI tools (like ollama, OpenAI, Chroma.db, HuggingFace, Dropbox, … you get the idea). I tend to browse existing cookbooks/workflows and then tailor them to my task.
I’ve recently learned about DSPy which I’m considering migrating to for building agentic workflows as it seems it may be better for tasks with deterministic outputs and customizing reasoning logic, which in particular for your counting task, would be a good fit.
In terms of what I use for free LLM use (a.k.a hosting it locally), it’s ollama. For me it’s been the most convenient wrapper of llama.cpp for running open-source models. I keep seeing use of vLLM as a CUDA-optimised alternative for batching, but it may not be a good fit for one-time tasks.
The last thing I’ll share is that I’ve found many of these bleeding-edge AI tools target GLIBC 2.31+ and I’ve mostly encounter HPCs running 2.28. So, containers will be your friend .
.