Course workspace
One account for your assignments, project, and API credit.
No credential needed for this local preview.
One account for your assignments, project, and API credit.
No credential needed for this local preview.
Homework 1
Due October 6, 11:59pm Eastern
You are an AI engineering consultant. You are getting hired by Bespoke Support, a company that helps businesses troubleshoot software. They are hiring you to build a tool, which will process their customer support conversations for public release on its help center.
Currently, the message dump that Bespoke Support want to share contains lots of private customer information. Publishing those details unredacted could violate a customer's privacy or even give someone access to their systems! You are here to help them.
For each case, Bespoke gives you a source conversation or set of notes, plus context from related messages and documentation. Your tool must replace private values with asterisks and leave every other character unchanged.
"The health check fails on my laptop, cedar-17," Sarah Miller wrote. "It started after I installed 2.4.0. Restarting the service did not help." Alex Chen replied: "This was a bug in 2.4.0. Upgrade to 2.4.1, then run curl localhost:8080/health. Please keep the response in the thread." "That fixed it," Sarah wrote. "The endpoint returns 200 now. You can reach me at sarah.miller@example.org if you need another test."
"The health check fails on my laptop, ********," ************ wrote. "It started after I installed 2.4.0. Restarting the service did not help." Alex Chen replied: "This was a bug in 2.4.0. Upgrade to 2.4.1, then run curl localhost:8080/health. Please keep the response in the thread." "That fixed it," ***** wrote. "The endpoint returns 200 now. You can reach me at ************************ if you need another test."
The context identifies (1) Sarah as a customer, (2) cedar-17 as her laptop, and (3) Alex as the product maintainer. Notice how in the redacted copy Sarah's details are hidden, but Alex's name and the fix are kept unredacted.
Implement redact(packet, llm) using Qwen3.5-9B, Qwen3.7 Flash, or both. In your function, decide which judgments to delegate to a model and which properties to enforce in code.
The package from Bespoke Support grants you the following resources:
We want to publish these troubleshooting conversations so other engineers can learn from them. Remove private information without losing the explanation of the problem and its fix. Return an exact copy with private text replaced by asterisks and every other character unchanged. We've provided examples and related context, and you can ask us to clarify our publication requirements.
Your job is to implement the function redact(packet, llm).
Each call receives a packet dictionary, consisting of: a source text to redact, related texts in context, and the publication_agreement.
We also supply llm, an object for calling the two allowed language models. Use llm.complete(prompt, model="qwen37f") to get a response as text, or choose model="qwen35" for the 9B model. It records token usage automatically.
For example:
packet = {
"source": """Sarah Miller: My laptop, cedar-17, fails its health check.
Alex Chen: Upgrade to 2.4.1, then restart the service.
Sarah: That fixed it. The endpoint returns 200 now.""",
"context": [
"Sarah Miller is the customer; cedar-17 is her laptop.",
"Alex Chen maintains the software product.",
],
"publication_agreement": "standard",
}
publication_agreement is either standard or volume-confidential. You can ask us what these agreements require through Client Q&A.
Context can arrive in any order. Use its contents, not the positions of texts in the list, to establish any necessary chronology. Your system should support up to 120,000 characters across source and context, and up to 160 context strings.
def redact(packet, llm):
return redacted_text
source but redacted, with exactly the same number of Unicode characters. At each position, either keep the original character or replace it with an ASCII asterisk *.In data/development-v1.json, only each row's case reaches your function. Other metadata like IDs, titles, groups, and labels will not be available for your system to use.
Related examples from the same troubleshooting case share a source_cluster. If you set aside examples for testing, keep each case together so you test on new cases, not another excerpt of one you've already used.
For your development, we share some expert labels with you on a small amount of troubleshooting records. The labels come in two lists: redact contains values we want concealed, whereas keep contains values you might think to conceal but which we actually want published!
Each item gives its start and end positions and the text. Keep labels are examples, not an inventory of everything that matters. All other text should remain untouched.
Our data compliance team can evaluate up to seven versions of your program on conversations you haven't seen and return a leaderboard score. Submit a version through the Submissions page. You can experiment with the supplied examples as often as you like without using these seven opportunities.
You may ask up to 30 questions about the requirements, one at a time. Keep in mind that the channel cannot help you solve particular examples. Due to our legal team's restrictions, we can only give you pre-approved answers; we appreciate your understanding. During the homework, only you see your questions and answers - please do not share this with other people.
Thank you,
The Bespoke Support team
Your leaderboard results contribute to your grade. After the deadline, we evaluate your selected versions on a separate, private set of examples that represents deployment.
Quality measures how well your publication matches Bespoke's requirements. The local checker reports missing redactions, unnecessary redactions, output validity, and token use. The client might be able to tell you more about how they weigh different errors.
| Model | Units / input token | Units / output token |
|---|---|---|
| Qwen3.5-9B | 1 | 3 |
| Qwen3.7 Flash | 4 | 12 |
The supplied llm records usage automatically. Evaluation counts every call, including retries and calls that produce no valid publication. Output tokens include reasoning tokens once; cached input tokens count at the full input rate.
Quality 60 and 100,000 mean token units yield net 56. These units create an engineering trade-off; they are not dollars. Your development calls are billed separately by OpenRouter.
| Component | Points |
|---|---|
| Report | 50 |
| Public leaderboard score (revealed via any-time submissions) | 35 |
| Private held-out score (revealed after deadline) | 35 |
The homework is out of 100 with up to 20 bonus points.
On the public leaderboard, credit increases proportionally from zero to 35/35 points at a net score of 20. Higher scores earn no additional public-set credit. Beyond that target, focus on whether your improvements generalize: the private evaluation rewards performance on new examples.
On the private held-out set, your points are 35 × your net score / 100. For example, a net score of 60 earns 21/35 points.
This estimate uses the same average cost for both evaluation sets.
See the Report page for the instructions, template, and separate report deadline.
As you work, keep notes on your hypotheses, what you tested, and the failures you observed. Record what changed your mind, including experiments that did not help. These notes will give you the evidence to write your report later, without having to reconstruct the process from memory.
Get your personal OpenRouter key from the Account tab and set it as OPENROUTER_API_KEY in your environment. Do not include it in your submission.
Open explore.ipynb in VS Code or Jupyter. It loads one example, shows its source and context, runs your program, and highlights the mistakes alongside the model usage.
You can iterate on the assignment in two ways. Either edit solution.py in your editor, or write your program in a notebook cell beginning with %%writefile solution.py. Running that cell writes your imports, helpers, and redact function to the file. In both workflows, the notebook tests that Python file. In general, you submit that file, not the notebook.
We strongly recommend working on one example at a time: skim it, run your program, understand some of its failures if any, and try some tweaks. Repeatedly changing the program in ad-hoc ways and rerunning the entire dataset can quickly consume your credit without teaching you much.Once a change makes sense, check it on a few other examples. The development set is all yours; you can do whatever you want on it, but your system must generalize to new unseen instances and, ideally, to a different capable LLM.
python3 -m hw1_utils run --limit 1 --out runs/first python3 -m hw1_utils check runs/first --details
Omit --limit to run all examples. Use --workers 4 to run four examples at a time in parallel, and provide a new output directory for each experiment. Predictions, labels, and usage are saved together.
The baseline in solution.py uses one plain model call. You are free to change it as you see fit. Edit that file as you work in the notebook; each run reloads it!
You can define helper functions in solution.py and call them from redact. You can also import helpers from your own Python files; include those files when you package your submission.
llm.complete(
prompt,
model="qwen37f", # or "qwen35"
system=None,
schema=None,
thinking=False,
temperature=None,
max_tokens=8192,
)
The call returns text. Specify model on every invocation to llm.complete. Notice that you can mix both models in one program. All calls share your execution budget.
You can use len(llm.calls) to count calls, llm.broker.tokens for completion tokens used, and llm.broker.cost_micros / 1_000_000 for the cost in dollars. These describe the current execution, both locally and during grading.
Grading runs Python 3.13 in an isolated environment with the standard library and no direct network access. Include your Python helper files, since third-party packages, threads, subprocesses, and runtime installation are not supported.
For each conversation, your program can make up to 32 model calls, spending at most $0.02 in API costs and producing at most 131,072 output tokens, including reasoning. These limits start fresh for the next example; the dollar cap is separate from the artificial token units in your score and your $35 account allowance. Your code has up to 15 minutes of its own running time. A model call can take up to 20 minutes, but the whole example, including all model waits, must finish within 60 minutes.
If your program stops because it exceeds a limit, that example receives zero, not automatically the entire submission. Model-service problems are handled separately; unresolved service or billing problems may be held for staff review.
The following are the settings applied by the supplied llm.complete, both locally and during grading. For Qwen3.5-9B, we use Qwen's general-task sampling recommendations: temperature 0.7 and top-p 0.8 without reasoning, or 1.0 and 0.95 with reasoning; top-k 20 and presence penalty 1.5 in both modes. For Flash, we leave sampling at the provider defaults unless you set temperature.
You can set temperature, thinking, and max_tokens on each call. OpenRouter chooses the serving provider within our price limits. You don't have to worry about this, but your usage log records which one answered.
By popular demand, Jev is available as an optional "decision" model. This is a less tested path and not as natural of a fit for HW1, so we do NOT recommend it for most students. Ignoring this optional model changes nothing about your code or score on the assignment. However, it seems that quite a few students really want to tinker with it, and we're happy to learn from the collective experience of the class. (We still recommend making at least one strong submission that does not use Jev.)
If you choose to try it, download the four optional helper files. The original starter package is unchanged; these helpers are a separate optional download. Replace hw1_utils/llm.py, hw1_utils/provider.py, and hw1_utils/costs.py, and add hw1_utils/decisions.py. Keep your own solution.py, notebooks, and other files. If you customized these helpers, merge the changes instead of overwriting them. No update is needed if you do not use Jev.
answers = llm.decide(
state={"message": "The app crashes when it starts."},
questions={"is_bug": {
"type": "noul",
"instructions": "Does the message report broken behavior?",
}},
)
probability = answers["is_bug"]["noul"]
state is the text or JSON evidence to evaluate. questions is a map of typed decisions; the returned answers use the same keys. The interface preserves Jev's choice, score, and noul question and answer formats. See the native API reference and OpenRouter decision endpoint.
Jev costs 1 unit per input token and 0 units per output token. Input usage includes the shared state and every question. It uses your existing OpenRouter key and shares the same dollar budget, call limit, and completion-token limit as Qwen. A batch of up to 128 questions counts as one call; reported output tokens still count toward the completion-token limit. Context must fit Jev's own model limit. Probabilities are model judgments, not guarantees.
python3 -m hw1_utils package
This saves a new dated ZIP in submissions/, containing a copy of your current solution.py. Functions defined in that file are already included. If your code imports your own helpers.py and formatting.py files, package them too:python3 -m hw1_utils package --include helpers.py formatting.pyUpload that ZIP on the Submissions page. You can download your own saved code from each submission later. Of course, editing your working files does not change a saved version.
You get up to seven scored submissions. Each completed result is saved. The leaderboard shows one row per student, containing your chosen public name (please anonymize it!) and highest net score. System names and submission history are private to your account, not shared with classmates, so feel free to describe whatever you need there so you can remember the difference between submissions.
Scoring runs in a queue and may take several hours near the deadline. Versions submitted before the deadline still count for automatic final selection if scoring finishes afterward.
Choose up to two saved versions for final evaluation. If you do not choose, we use your two highest leaderboard scores, with earlier submissions breaking ties. We grade each version separately and keep the better total.
Your $35 OpenRouter allowance covers local experiments and leaderboard evaluations, and we expect that most students will not even need most of that. Remaining credit appears at the top of every page; you can also check it with python3 -m hw1_utils balance. Ask the teaching team if you need more. Final grading is funded separately.
Please submit a report on your system on Gradescope by Oct 9th, 11:59pm Eastern. The system deadline remains Oct 6th. Remember that we already have your code through your leaderboard submissions, but make sure you select TWO final systems on the portal. The higher scoring of the two will be used for held-out scoring.
The context of this report is that a manager at Bespoke thinks they should have just asked a frontier LLM coding agent "please write a redaction script". But given their lack of expertise in AI engineering, we suspect that Bespoke may not know how to guide that coding agent or how to ultimately tell whether the resulting system is reliable or maintainable enough for their use. By now, you've spent plenty of time on these questions. Write an informal scientific report in your voice explaining to Bespoke (as well as others building systems like this) what you learned and what building this system required.
Use the following single-column report template: a signed cover page, then 1.5-3 pages of main text. Put your name or other identifying information on the cover ONLY so we can remove it before grading. You should write primarily about the work you did before the Oct 6th system submission deadline, but you may add a paragraph or two reflecting on the Oct 7th class discussion if you wish. In your report, please make sure to discuss:
You are allowed to use AI tools to collect data for your report and for help with preparing the report itself. That said, the goal of this report is to communicate your role (and your ownership of your system) to Bespoke. Hence, the report should reflect your experience building the system and your voice and, in particular, every statement in your report must reflect judgments that you formed yourself from observing the underlying facts.
Grading: We expect a reasonable report to earn at least 45/50 points, or possibly higher. Reports we assess to be thoughtful or informative at a deeper level will earn full credit and could possibly be eligible for further bonus.
Prepare customer support conversations for a public help center.
Not released
Not released
Use this key for local experiments and leaderboard evaluations. Client Q&A does not use your API credit. Keep your key private and out of your submitted code.
Your $35 allowance covers local experiments and leaderboard evaluations. The teaching team pays for final grading separately. Contact us if you need more credit.
Your work
Your best public and private results count separately. Report grades are separate.
| Selected system | Public NET | Private NET | Private quality | Private cost |
|---|
Completed submissions appear on the leaderboard automatically. Packaging instructions
| Final | System name | Status | Net score | Quality | Cost penalty | API spend |
|---|
Your scored versions will appear here.
Keep the two highest-scoring versions, or choose up to two yourself.
HW1 / Redaction for a Public Help Center
This result contributes up to 35 points. Your final grade also includes the report and a separate held-out evaluation.
| Rank | Name | Net score | Quality | Cost |
|---|
Sol is here for comparison, ranked by quality. Its cost is shown relative to the starter.
No matching names.
We included three simple baselines to give you a feel for the scores different systems can achieve. There is room to do better, and we encourage you to try. But be careful about submitting repeatedly to try to game the public leaderboard. The real challenge is to build a system that generalizes to unseen and potentially domain-shifted data.
Indeed, you can earn the full 35% of your homework grade allocated to the public leaderboard if you achieve a net score of 20. Beyond that, improvements increase your grade only when they carry over to the private held-out examples, where higher net scores continue to earn more credit all the way to 100. With full report and public-leaderboard credit, a private net score of 100 brings your homework grade to 120%, including 20 percentage points of bonus credit. Evaluation details
Your questions and answers stay private from other students until the deadline. After the assignment is due, we may anonymously compare the questions asked while building stronger and weaker systems.
No matching questions or replies.