Numerous blog posts and YouTube videos discuss how good Claude Code is as an SWE agent. However, I would like to discuss my experience using it for data analysis. And my conclusion is that it’s revolutionary and has completely changed my way of working.
I’ll share my setups and the lessons I learned. But if you haven’t used Claude Code before, the best thing you can do is stop reading this and give it a try.
I’m a researcher. In my day-to-day work, I need to write (mostly) Python code to collect and process data, gain insights, and then turn them into visually appealing figures and tables. Finally, I write papers about the findings.
A significant portion of my coding tasks involves building data pipelines and exploring patterns in the data (i.e., the dirty work), which can be tedious and time-consuming.
With the development of various AI agents, I started to wonder what a data analysis agent would look like and even considered building one myself. Then I played with Claude Code (which I’ll refer to as CC from here on), and I realized that it was exactly what I was looking for.
In the past few weeks, I have been using CC to perform the data analysis for a paper I’m working on. It was a fascinating journey. Ultimately, I believe it generated over 98% of the code. And I think I will never go back to my old working routine again.
I’m using a combination of the following for coding tasks:
Claude Code: This is my main coding agent.
Lazygit: Git is your best friend for vibe coding (more on this later), and lazygit is a great terminal UI for Git.
Cursor: I primarily use it for reviewing the code generated by CC and making minor modifications as needed. The tab completion feature is still super handy.
Good old Jupyter Notebook: Although Cursor supports notebooks now, the experience is just not good. So I would still open the notebook in my browser.
I have two monitors so that I would have the Cursor in one and a browser window (with Jupyter Notebooks) in the other. Moreover, I would use my iPad as the third screen with CC in it. The screen size of the iPad is actually perfect for CC. Although one can run CC directly in Cursor, I prefer running it in the terminal.
There are many other coding agents, and I have tried the following:
OpenAI Codex CLI: I only tried it briefly; somehow, it sucks so bad.
Gemini CLI: Similar to Codex, not even close to CC. But I think it has a lot of potential. Guess we will see.
Cursor agent: It’s decent, but not at the CC level. Additionally, with the new pricing strategy for the Pro plan, the included API calls are pretty limited.
I haven’t tried others, but based on others' feedback, I’m convinced that CC is currently the best in terms of coding ability.
CC is also great due to the incredible value it provides, given its price. I'm on the Pro plan, which only costs $17 a month if you pay yearly. It's rate-limited but enough for me (at least for now). And you can easily burn over $100 worth of API calls each month using the Pro subscription (you can use the ccusage tool to monitor your usage). When I reach the rate limit but need to complete a task, I switch to the bring-your-own-key mode. Opus 4 is not available in the Pro plan, but I find Sonnet 4 more than competent.
This should work for all VSCode-based editors. See instructions here. It allows CC to be aware of which file you are focusing on in the editor. You can select specific lines of code to highlight them. The integration works even when you are running CC in a separate terminal window.
Hooks is a new feature that was recently added. It allows you to register bash commands at different points in time during CC’s actions. There are many cool things you can do with it, but I mainly use it to send me notifications when CC finishes a task or needs my input.
Below is my setting:
| "hooks": { | |
| "Notification": [ | |
| { | |
| "matcher": "", | |
| "hooks": [ | |
| { | |
| "type": "command", | |
| "command": "jq -r '.message' | xargs -I{} terminal-notifier -message \"{}\" -title \"Claude Code\" -sound default" | |
| } | |
| ] | |
| } | |
| ], | |
| "Stop": [ | |
| { | |
| "matcher": "", | |
| "hooks": [ | |
| { | |
| "type": "command", | |
| "command": "terminal-notifier -message \"✅ Task finished.\" -title \"Claude Code\" -sound default" | |
| } | |
| ] | |
| } | |
| ] | |
| } |
It only works on macOS and requires the terminal-notifier tool.
CC can be further customized. I would recommend reading the official documentation for Claude Code. It’s not very lengthy, but it can help you understand all the functionalities and settings. The tool also evolves rapidly, making the official documentation the primary source for the most up-to-date information.
Although CC is magical, it won’t always produce the output that you want, and it makes mistakes all the time. To me, working with CC effectively is all about aligning CC with your expectations and identifying and correcting any errors. Below are some lessons I learned.
Git can help you understand what changes CC made, and you can roll back easily when things are messed up.
Good news is that CC can leverage Git as well. You could just tell it to create a new branch or make a commit for the changes it just made.
I’m not sure if it’s due to the limited context window, but it’s very difficult for CC to write the entire project in one run correctly. A more effective approach for me is to make a plan for your project first, then focus on smaller tasks one at a time. The plan mode (shift+tab twice to switch to it) is perfect for creating plans.
I would usually ask CC to put the plan into a Markdown file. This gives me a chance to modify the details. Also, the Markdown file serves as a persistent context that you can refer to later, even if you clean the conversation history. The Markdown file also serves as documentation for the component.
A typical project of mine would include the following components:
Data cleaning
Analysis for research question 1
Analysis for research question 2
Output generation (figures and tables)
I would create a grant plan describing the basic logic and the expected inputs and outputs for each component. Then, for each individual component, I would have a more specific plan that includes different steps. Each step would correspond to a script.
With all the plans in place, I would ask CC to generate the scripts one at a time. After each script is generated, I would review the code and check the output to ensure it meets the requirements.
Yes, I still need to review and validate the results and modify the code when needed (this is why I still need an editor like Cursor). CC is great, but it often makes mistakes or misunderstands my requirements. Since I’ll be taking responsibility for the final results, reviewing the code is necessary. Fixing the errors at each step along the way also helps avoid the cascade of errors in the pipeline and makes sure you are in control the whole time.
CC is an agentic framework, which means it knows when to use tools to achieve its goals. By default, it could use various CLI-based tools to perform tasks such as searching and editing. As mentioned earlier, CC can also utilize Git to manage code versions. This capability allows CC to explore what’s available in your project and leverage all the information as context.
The nice thing with CC is that you can give it more tools!
For example, I would suggest installing GitHub’s CLI tool. This way, CC can create pull requests or check the issues on your behalf.
For my data analysis project, I use Parquet files in the pipeline. Parquet files are not easily readable in the CLI. So I installed parquet-tools. Then, in the CLAUDE.md file, I added the following instructions:
You are working in an environment where the tool parquet-tools is installed.
The tool can be used to view and inspect the parquet files.
Use
parquet-tools show -n 10 <path_to_parquet_file>to view the first 10 rows of the parquet file. Always specify the top n rows to view, otherwise the tool will show all rows, which is not helpful.
Use
parquet-tools inspect <path_to_parquet_file>And just like that, CC can now easily check the schema and content of Parquet files. It’s very useful in at least two cases. When CC needs to create an analysis plan or script, it first reads the input files to gain a sense of the data and schema, and then determines the correct input files. When CC finishes a script, it actually runs it and checks the output to ensure it meets the requirements. The magic part is that CC would do these automatically without explicit instructions.
Another way to augment CC is through MCP servers. Personally, I haven’t found any super helpful MCP servers for my projects. Plus, if I need a new functionality, I prefer to turn it into a bash command and provide it to CC.
Context is perhaps the most critical factor for getting CC to generate the output you want. In other words, if CC produces something that deviates from your expectation, it’s probably because you haven’t provided the complete or correct context.
In a data analysis project, context encompasses various elements, including the plans, the input/output data files, and the requirements for each script. CC is very clever in the sense that it would actively explore the environment to gather the context it needs. But it may fail from time to time and needs your guidance.
One thing you need to do is manage the current context window. For example, Sonnet 4 has a 200K context window. CC’s context grows rapidly as you continue to have conversations with it and as it executes its tasks. You want to make sure the current context window only includes information related to your specific task, and nothing else. Therefore, you may need to clear the conversation history periodically to remove irrelevant information. When you start a new conversation, you need to leverage the data, such as the plan from earlier, to populate the current context window. You can also mention files or select specific code blocks in your editor to provide pointers to CC.
I’m not sure how to accurately describe this tip. The idea is that since it’s relatively inexpensive to generate code to explore the data, you should fully leverage it.
For example, you could ask CC to produce summary statistics at every step in the pipeline, so you have a good sense of everything. In fact, CC would do this automatically most of the time, so I don’t even need to ask.
Similarly, you could ask CC to try different ways of analyzing or presenting the data for you to explore and choose from. You could also tell CC to convert the outputs into a nice HTML page, making it easier for you to examine or share with your colleagues.
These tasks can be pretty time-consuming if performed manually. However, with CC, it’s now much easier.
For me, the goal is to increase productivity, not to use CC on everything. This means that it’s essential to understand when to use CC and when to do it manually.
For instance, the data exploration tasks are perfect for CC. I could literally ask CC to “do data exploration” without even specifying exactly how, and it would provide reasonable outcomes based on its understanding of the data.
However, when it comes to producing figures for publication, where precision is key, I found it easier to do it myself. I suppose it’s because many tweaks are needed to achieve the desired outcome, and it’s much easier to edit myself than to describe them to CC.
As a disclaimer, my fondness for CC is based on the landscape in mid-2025. Last year, around the same time, my favorite was Cursor. So who knows what would happen next year?
That said, I think some trends are clear. We would have more capable AI agents, and CC provides a perfect template for them. So many of the tips included here should apply to other AI agents as well.
No posts