Course workspace
One account for your assignments, project, and API credit.
Instructor preview
No credential needed for this local preview.
AI Engineering6.S978 / Fall 2026
AI Engineering6.S978 / Fall 2026
One account for your assignments, project, and API credit.
No credential needed for this local preview.
Homework 1
Due October 6, 11:59 p.m. Eastern
You are an AI engineering consultant. You are getting hired by Bespoke Support, a company that helps businesses troubleshoot software. They are hiring you to build a tool, which will process their customer support conversations for public release on its help center.
Currently, the message dump that Bespoke Support want to share contains lots of private customer information. Publishing those details unredacted could violate a customer's privacy or even give someone access to their systems! You are here to help them.
For each case, Bespoke gives you a source conversation or set of notes, plus context from related messages and documentation. Your tool must replace private values with asterisks and leave every other character unchanged.
"The health check fails on my laptop, cedar-17," Sarah Miller wrote. "It started after I installed 2.4.0. Restarting the service did not help." Alex Chen replied: "This was a bug in 2.4.0. Upgrade to 2.4.1, then run curl localhost:8080/health. Please keep the response in the thread." "That fixed it," Sarah wrote. "The endpoint returns 200 now. You can reach me at sarah.miller@example.org if you need another test."
"The health check fails on my laptop, ********," ************ wrote. "It started after I installed 2.4.0. Restarting the service did not help." Alex Chen replied: "This was a bug in 2.4.0. Upgrade to 2.4.1, then run curl localhost:8080/health. Please keep the response in the thread." "That fixed it," ***** wrote. "The endpoint returns 200 now. You can reach me at ************************ if you need another test."
The context identifies (1) Sarah as a customer, (2) cedar-17 as her laptop, and (3) Alex as the product maintainer. Notice how in the redacted copy Sarah's details are hidden, but Alex's name and the fix are kept unredacted.
Implement redact(packet, llm) using Qwen3.5-9B, Qwen3.7 Flash, or both. In your function, decide which judgments to delegate to a model and which properties to enforce in code.
The package from Bespoke Support grants you the following resources:
We want to publish these troubleshooting conversations so other engineers can learn from them. Remove private information without losing the explanation of the problem and its fix. Return an exact copy with private text replaced by asterisks and every other character unchanged. We've provided examples and related context, and you can ask us to clarify our publication requirements.
Your job is to implement the function redact(packet, llm).
Each call receives a packet dictionary, consisting of: a source text to redact, related texts in context, and the publication_agreement.
We also supply llm, an object for calling the two allowed language models. Use llm.complete(prompt, model="qwen37f") to get a response as text, or choose model="qwen35" for the 9B model. It records token usage automatically.
For example:
packet = {
"source": """Sarah Miller: My laptop, cedar-17, fails its health check.
Alex Chen: Upgrade to 2.4.1, then restart the service.
Sarah: That fixed it. The endpoint returns 200 now.""",
"context": [
"Sarah Miller is the customer; cedar-17 is her laptop.",
"Alex Chen maintains the software product.",
],
"publication_agreement": "standard",
}
publication_agreement is either standard or volume-confidential. You can ask us what these agreements require through Client Q&A.
Context can arrive in any order. Use its contents, not the positions of texts in the list, to establish any necessary chronology. Your system should support up to 120,000 characters across source and context, and up to 160 context strings.
def redact(packet, llm):
return redacted_text
source but redacted, with exactly the same number of Unicode characters. At each position, either keep the original character or replace it with an ASCII asterisk *.In data/development-v1.json, only each row's case reaches your function. Other metadata like IDs, titles, groups, and labels will not be available for your system to use.
Related examples from the same troubleshooting case share a source_cluster. If you set aside examples for testing, keep each case together so you test on new cases, not another excerpt of one you've already used.
For your development, we share some expert labels with you on a small amount of troubleshooting records. The labels come in two lists: redact contains values we want concealed, whereas keep contains values you might think to conceal but which we actually want published!
Each item gives its start and end positions and the text. Keep labels are examples, not an inventory of everything that matters. All other text should remain untouched.
Our data compliance team can evaluate up to seven versions of your program on conversations you haven't seen and return a leaderboard score. Submit a version through the Submissions page. You can experiment with the supplied examples as often as you like without using these seven opportunities.
You may ask up to 30 questions about the requirements, one at a time. The channel MIGHT close up to 48 hours before the deadline. Keep in mind that the channel cannot help you solve particular examples. Due to our legal team's restrictions, we can only give you pre-approved answers; we appreciate your understanding. During the homework, only you see your questions and answers - please do not share this with other people.
Thank you,
The Bespoke Support team
Your leaderboard results contribute to your grade. After the deadline, we evaluate your selected versions on a separate, private set of examples that represents deployment.
Quality measures how well your publication matches Bespoke's requirements. The local checker reports missing redactions, unnecessary redactions, output validity, and token use. Ask the client how it weighs these mistakes; the numerical quality formula is published with the grades.
| Model | Units / input token | Units / output token |
|---|---|---|
| Qwen3.5-9B | 1 | 3 |
| Qwen3.7 Flash | 4 | 12 |
The supplied llm records usage automatically. Evaluation counts every call, including retries and calls that produce no valid publication. Output tokens include reasoning tokens once; cached input tokens count at the full input rate.
Quality 60 and 100,000 mean token units yield net 56. These units create an engineering trade-off; they are not dollars. Your development calls are billed separately by OpenRouter.
| Component | Points |
|---|---|
| Report | 50 |
| Public leaderboard score (revealed via any-time submissions) | 35 |
| Private held-out score (revealed after deadline) | 35 |
The homework is out of 100 with up to 20 bonus points.
On the public leaderboard, credit increases proportionally from zero to 35/35 points at a net score of 20. Higher scores earn no additional public-set credit. Beyond that target, focus on whether your improvements generalize: the private evaluation rewards performance on new examples.
On the private held-out set, your points are 35 × your net score / 100. For example, a net score of 60 earns 21/35 points.
This estimate uses the same average cost for both evaluation sets.
Report instructions will follow. You do not need a report for leaderboard submissions.
As you work, keep notes on your hypotheses, what you tested, and the failures you observed. Record what changed your mind, including experiments that did not help. These notes will give you the evidence to write your report later, without having to reconstruct the process from memory.
Get your personal OpenRouter key from the Account tab and set it as OPENROUTER_API_KEY in your environment. Do not include it in your submission.
Open explore.ipynb in VS Code or Jupyter. It loads one example, shows its source and context, runs your program, and highlights the mistakes alongside the model usage.
You can iterate on the assignment in two ways. Either edit solution.py in your editor, or write your program in a notebook cell beginning with %%writefile solution.py. Running that cell writes your imports, helpers, and redact function to the file. In both workflows, the notebook tests that Python file. In general, you submit that file, not the notebook.
We strongly recommend working on one example at a time: skim it, run your program, understand some of its failures if any, and try some tweaks. Repeatedly changing the program in ad-hoc ways and rerunning the entire dataset can quickly consume your credit without teaching you much.Once a change makes sense, check it on a few other examples. The development set is all yours; you can do whatever you want on it, but your system must generalize to new unseen instances and, ideally, to a different capable LLM.
python3 -m hw1_utils run --limit 1 --out runs/first python3 -m hw1_utils check runs/first --details
Omit --limit to run all examples. Use --workers 4 to run four examples at a time in parallel, and provide a new output directory for each experiment. Predictions, labels, and usage are saved together.
The baseline in solution.py uses one plain model call. You are free to change it as you see fit. Edit that file as you work in the notebook; each run reloads it!
You can define helper functions in solution.py and call them from redact. You can also import helpers from your own Python files; include those files when you package your submission.
llm.complete(
prompt,
model="qwen37f", # or "qwen35"
system=None,
schema=None,
thinking=False,
temperature=None,
max_tokens=8192,
)
The call returns text. Specify model on every invocation to llm.complete. Notice that you can mix both models in one program. All calls share your execution budget.
Grading runs Python 3.13 in an isolated environment with the standard library and no direct network access. Include your Python helper files, since third-party packages, threads, subprocesses, and runtime installation are not supported.
The following are the settings applied by the supplied llm.complete, both locally and during grading. For Qwen3.5-9B, we use Qwen's general-task sampling recommendations: temperature 0.7 and top-p 0.8 without reasoning, or 1.0 and 0.95 with reasoning; top-k 20 and presence penalty 1.5 in both modes. For Flash, we leave sampling at the provider defaults unless you set temperature.
You can set temperature, thinking, and max_tokens on each call. OpenRouter chooses the serving provider within our price limits. You don't have to worry about this, but your usage log records which one answered.
python3 -m hw1_utils package
This saves a new dated ZIP in submissions/, containing a copy of your current solution.py. Functions defined in that file are already included. If your code imports your own helpers.py and formatting.py files, package them too:python3 -m hw1_utils package --include helpers.py formatting.pyUpload that ZIP on the Submissions page. You can download your own saved code from each submission later. Of course, editing your working files does not change a saved version.
You get up to seven scored submissions. Each completed result is saved. The leaderboard shows one row per student, containing your chosen public name (please anonymize it!) and highest net score. System names and submission history are private to your account, not shared with classmates, so feel free to describe whatever you need there so you can remember the difference between submissions.
Scoring runs in a queue and may take several hours near the deadline. Versions submitted before the deadline still count for automatic final selection if scoring finishes afterward.
Choose up to two saved versions for final evaluation. If you do not choose, we use your two highest leaderboard scores, with earlier submissions breaking ties. We grade each version separately and keep the better total.
Your $35 OpenRouter allowance covers local experiments and leaderboard evaluations, and we expect that most students will not even need most of that. Remaining credit appears at the top of every page; you can also check it with python3 -m hw1_utils balance. Ask the teaching team if you need more. Final grading is funded separately.
Prepare customer support conversations for a public help center.
Not released
Not released
Use this key for local experiments and leaderboard evaluations. Client Q&A does not use your API credit. Keep your key private and out of your submitted code.
Your $35 allowance covers these three uses. The teaching team pays for final grading separately. Contact us if you need more credit.
Your work
Completed submissions appear on the leaderboard automatically. Packaging instructions
| Final | System name | Status | Net score | Quality | Cost penalty | API spend |
|---|
Your scored versions will appear here.
Keep the two highest-scoring versions, or choose up to two yourself.
HW1 / Redaction for a Public Help Center
This result contributes up to 35 points. Your final grade also includes the report and a separate held-out evaluation.
| Rank | Name | Net score | Quality | Cost penalty |
|---|
No matching names.
One best score per student, ranked alongside the baselines. Evaluation details
Your questions and answers stay private from other students until the deadline. After the assignment is due, we may anonymously compare the questions asked while building stronger and weaker systems.
No matching questions or replies.