AdmitYogi Essays

Context

Crimson Education’s essay review service supports high school students in writing their college application essays. The team delivering this service are world-class admission and essay writing experts which means students receive a high quality product—but the pricing is steep. A single essay review by that team costs as much as US$85, and comes with a four day turnaround time.

Part of Crimson’s public mission is to “equalize the […] admissions playing field.” To truly achieve this goal, it was clear that we had to find a way to scale access to essay review services by providing it at a much cheaper price point.

Around this point in time, OpenAI had made API access to their GPT-3.5 Turbo and GPT-4 language models generally available to developers.

We therefore had the ideal circumstances for innovation. The combination of in-house essay review expertise, a big corpus of real admission essays, the GPT family of language models, and my technical expertise were all the ingredients necessary.

Solving AI essay review

I was asked by Crimson’s CPO to come in and lead the technical implementation effort and provide guidance to our PM. My pre-existing expertise with supervised learning approaches and generative AI tools coupled with my track record of getting big projects out the door fast meant I was the ideal candidate to drive execution.

I worked with our PM to carve up the project. She would be responsible for tackling high-level aspects of the product such as the product design and positioning, while I would take charge of all implementation details. I brought with me a junior software engineer from my team at Crimson Global Academy to assist in frontend development, and a member of my AI Unit at Crimson to assist with building out the essay review functionality.

I knew that by far the hardest component of the project was the essay review system. The project was never going to die because of a failure to implement a frontend design, but it would absolutely die if it wasn’t able to fulfill its core value proposition.

Therefore, the first step in the process—and the longest and most complicated—was to immediately start work on that piece. Working closely with my fellow AI Engineer, our PM, and Crimson’s internal essay review team, we were able to collect a text corpus to work with.

That corpus consisted of real essays the essay review team had been sent by students, alongside the reviews they had sent back. This dataset was crucially important for the development of our overall processing pipeline, as it gave us real examples to work backwards from. I spent a lot of time performing an exploratory data analysis on this corpus in order to better understand the content.

Having performed a deep dive into these real specimens and discussing them in depth with our subject matter experts, we were in a position to make forward progress on the technical implementation.

Broadly, we tackled this problem in three different parts:

1. What kind of essays are there? We identified a wide variety of discrete essay types, like the “personal statement” and “academic interest” essays.
2. How do we break down the review process? A weak essay fundamentally needs different feedback from a strong essay. A weak essay, for instance, usually lacks a clear central theme whereas a strong essay always has one. We settled on a three-step process.
3. What kind of recommendations do reviewers make? At the third step of our review process—where we’ve validated the essay’s baseline quality—we then proceed to make more “tactical” recommendations. We analyzed hundreds of real essay reviews to create a taxonomy of these “tactics.”

These three elements gave us the tools we needed to reason about the problem space. We could, for instance, take a very concrete suggestion regarding the placement of a paragraph and reason about that suggestion in terms of whether it was a “signposting” or “cohesion” recommendation.

Finishing off the MVP

While the essay review was definitely the “scary bit” of the Dr Ivy project, there was far more to deal with than just the AI. There’s a pretty big list of things you need before a product is ready for the limelight. Here’s a list of other deliverables we shipped over the 3 months we spent building out Dr Ivy:

• User accounts system supporting both social logins such as Google, and a “Sign in with Crimson” feature for existing Crimson students.
• A mobile responsive UI to accommodate users on the go, or who don’t have access to a full-sized computer.
• Integration with Crimson’s data lake. We set up ETL jobs to sync up Dr Ivy’s PostgreSQL database with Crimson’s Snowflake data lake, to support analytics and other BI use cases.
• A gorgeous design system featuring crisp typography and liberal use of color to keep students engaged.
• User onboarding flow to capture demographic information about our users.
• A rich text editor using Slate as the foundation, housing a variety of custom plugins.
• APM and log management through Datadog, to fit in nicely with the rest of Crimson’s observability infrastructure.
• “Guest mode” which allows anonymous users to “try before they buy,” by giving them one essay review before being prompted to create an account.

All in all, we went cycled through three different major redesigns and ran two small-scale marketing campaigns before settling on the version of the product we shipped in August 2023. This represented a tremendous amount of work over a very short time period. A bias for action, commitment to keeping overheads low, and constant customer interviews all through the development process were essential components of our success.

Handover

At this point in time the Dr Ivy product was stable, and we’d had plenty of real students use the tool and provide us with valuable feedback. We’d solved the problem of reviewing US college admission essays, but we didn’t want to stop there. There were a number of exciting ideas on the roadmap for our Dr Ivy product.

After leading the engineering efforts and confirming the product’s viability, it was time to replace myself so I could move over to other high impact projects. In the short term the engineer I hired for the job would be responsible for implementing monetization and integrating the product into our newly-acquired AdmitYogi business.

I interviewed a few different engineers, and ultimately settled on Kriss Gardner—co-founder of Tenxbase. After cutting a deal, I showed Kriss around the codebase and provided as-needed technical support for the next two months as he worked on shipping the next set of deliverables for the product.

Business impact

Since the handover process finished, more than 24,000 essays have been reviewed by the AdmitYogi Essays product. Under today’s pricing model, that is six figures of revenue over the past 4 months following launch.

Crimson has also worked on an “Essay Hub” feature inside the “Crimson App.” This leverages APIs exposed by AdmitYogi to offer the same high-quality essay review feedback provided by AdmitYogi Essays from inside the Crimson App.

The release of this feature will mean that Crimson’s students going forward will be able to benefit from a one-stop-shop for all things related to essay writing. We expect to save thousands of hours of staff time per year by removing the need for students to manage their own Google Docs, and from automating the bulk of the essay review process.

Key takeaways

The biggest single factor driving the success of this project was the contact we had with subject matter experts. Crimson’s in-house essay review team played a pivotal role in helping design and build the framework for the AI essay review system we ultimately settled on. LLMs are a tremendously powerful and flexible technology but for use cases like AdmitYogi Essays, domain experts are still a requirement.

One layer up, I worked hard to instill an extreme bias for action in the small team we had working on this project. I firmly believe that you’re better off getting things done and shipping something rather than getting stuck in the weeds. The implementation phase of this product saw us cycle through multiple different product ideas, and this worked out well.

Even though this was—strictly speaking—a source of rework, it was a net positive. We were able to iterate on these ideas extremely quickly and were able to make subsequent iterations with ever-increasing confidence as we were always working off real feedback acquired during user testing. It’s important to remember that rework only slows you down when the cost of that rework is greater than the value you extracted from the previous iteration.

The AdmitYogi Essays product continues to grow and deliver returns to this day.