The code and data behind xeiaso.net
notes: Add CookTok note
Signed-off-by: Xe Iaso <me@xeiaso.net>
+216
+216
lume/src/notes/2024/cooktok.mdx
+216
lume/src/notes/2024/cooktok.mdx
···
1
+
---
2
+
title: "Making home cooking easier with Cooktok"
3
+
desc: "Maybe langle mangles can be used for good"
4
+
date: 2024-10-14
5
+
hero:
6
+
ai: "Photo by Xe Iaso, Canon EOS R6 mk. ii, Canon 16mm f/2.8 STM, f/8, iso 200"
7
+
file: gatineau-park-road
8
+
prompt: "A concrete road bifurcating a nature park, the sun is overhead making the sky a light blue. Shadows from the trees coat the road."
9
+
---
10
+
11
+
I've been cooking more with my husband, and in the process we're trying to branch out and make things we've never had before. Since my last job made me learn how to use TikTok, I found a bunch of easy to cook meals that I've wanted to make. There's just one problem, and I think it'll be obvious if you watch this video:
12
+
13
+
<blockquote
14
+
class="tiktok-embed"
15
+
cite="https://www.tiktok.com/@cookinginthemidwest/video/7320234168615505195"
16
+
data-video-id="7320234168615505195"
17
+
style="max-width: 605px;min-width: 325px;"
18
+
>
19
+
{" "}
20
+
<section>
21
+
{" "}
22
+
<a
23
+
target="_blank"
24
+
title="@cookinginthemidwest"
25
+
href="https://www.tiktok.com/@cookinginthemidwest?refer=embed"
26
+
>
27
+
@cookinginthemidwest
28
+
</a>
29
+
Kielbasa Sheet Pan Meal is easy to make and one the whole family enjoys!
30
+
INGREDIENTS (2) 12 oz packs of Beef Kielbasa 3 large potatoes 2 green bell
31
+
peppers Red onion 2 tbsp olive oil Salt, pepper, garlic powder, paprika
32
+
Buffalo Wild Wings Asian Zing INSTRUCTIONS Slice the kielbasa into thin
33
+
rounds. Cut the peppers and onions into pieces all about the same size.
34
+
Peel, wash, and chop the potatoes. Be sure to cut the potatoes into smaller
35
+
cubes. Add the potatoes to a large bowl and cover in olive oil. Season with
36
+
some salt, pepper, garlic powder, and paprika. Toss the potatoes well to
37
+
making sure they are well coated in seasoning. Add the chopped peppers and
38
+
onions and add a little bit more of the seasoning and oil. Toss/stir
39
+
everything together well. Spray a large sheet pan with cooking spray. Pour
40
+
the potatoes and veggies out onto the sheet pan. Top with the sliced
41
+
kielbasa. Bake in the oven at 400 degrees for 30-40 minutes. I suggest
42
+
checking them around 30 minutes and cooking for the additional time if
43
+
needed. Mine cooked for 40 minutes. Top with Buffalo Wild Wing’s Asian Zing
44
+
sauce and serve! The Asian Zing sauce is spicy, so I left a little bit
45
+
without it for my kids and they ate theirs with BBQ sauce.
46
+
<a
47
+
title="sheetpanmeal"
48
+
target="_blank"
49
+
href="https://www.tiktok.com/tag/sheetpanmeal?refer=embed"
50
+
>
51
+
#sheetpanmeal
52
+
</a> <a
53
+
title="kielbasa"
54
+
target="_blank"
55
+
href="https://www.tiktok.com/tag/kielbasa?refer=embed"
56
+
>
57
+
#kielbasa
58
+
</a> <a
59
+
title="easyrecipe"
60
+
target="_blank"
61
+
href="https://www.tiktok.com/tag/easyrecipe?refer=embed"
62
+
>
63
+
#easyrecipe
64
+
</a> <a
65
+
title="recipe"
66
+
target="_blank"
67
+
href="https://www.tiktok.com/tag/recipe?refer=embed"
68
+
>
69
+
#recipe
70
+
</a> <a
71
+
title="buffalowildwings"
72
+
target="_blank"
73
+
href="https://www.tiktok.com/tag/buffalowildwings?refer=embed"
74
+
>
75
+
#buffalowildwings
76
+
</a> <a
77
+
title="bwwsauce"
78
+
target="_blank"
79
+
href="https://www.tiktok.com/tag/bwwsauce?refer=embed"
80
+
>
81
+
#bwwsauce
82
+
</a> <a
83
+
title="asianzing"
84
+
target="_blank"
85
+
href="https://www.tiktok.com/tag/asianzing?refer=embed"
86
+
>
87
+
#asianzing
88
+
</a> <a
89
+
target="_blank"
90
+
title="♬ original sound - Cooking in the Midwest"
91
+
href="https://www.tiktok.com/music/original-sound-7320234231135816494?refer=embed"
92
+
>
93
+
♬ original sound - Cooking in the Midwest
94
+
</a>{" "}
95
+
</section>{" "}
96
+
</blockquote> <script async src="https://www.tiktok.com/embed.js"></script>
97
+
98
+
Here's the description of that video with the recipe included:
99
+
100
+
> Kielbasa Sheet Pan Meal is easy to make and one the whole family enjoys! INGREDIENTS (2) 12 oz packs of Beef Kielbasa 3 large potatoes 2 green bell peppers Red onion 2 tbsp olive oil Salt, pepper, garlic powder, paprika Buffalo Wild Wings Asian Zing INSTRUCTIONS Slice the kielbasa into thin rounds. Cut the peppers and onions into pieces all about the same size. Peel, wash, and chop the potatoes. Be sure to cut the potatoes into smaller cubes. Add the potatoes to a large bowl and cover in olive oil. Season with some salt, pepper, garlic powder, and paprika. Toss the potatoes well to making sure they are well coated in seasoning. Add the chopped peppers and onions and add a little bit more of the seasoning and oil. Toss/stir everything together well. Spray a large sheet pan with cooking spray. Pour the potatoes and veggies out onto the sheet pan. Top with the sliced kielbasa. Bake in the oven at 400 degrees for 30-40 minutes. I suggest checking them around 30 minutes and cooking for the additional time if needed. Mine cooked for 40 minutes. Top with Buffalo Wild Wing’s Asian Zing sauce and serve! The Asian Zing sauce is spicy, so I left a little bit without it for my kids and they ate theirs with BBQ sauce.
101
+
102
+
This is completely unreadable. The really confusing part is that in the app this isn't that bad:
103
+
104
+
<img
105
+
src="https://cdn.xeiaso.net/file/christine-static/notes/2024/cooktok/IMG_1571.jpg"
106
+
width={300}
107
+
className="mx-auto"
108
+
/>
109
+
110
+
When I made this, [Hermes 3](https://nousresearch.com/hermes3/) by Nous Research had just been released and one feature of the model piqued my curiosity: [JSON schema based prompting](https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-405B#prompt-format-for-json-mode--structured-outputs). This isn't documented very well anywhere, but in order to do it, you need to pass a system prompt like this:
111
+
112
+
<div className="mx-auto" style="max-width: 78ch;">
113
+
```
114
+
You are a helpful assistant that answers in JSON.
115
+
Here's the json schema you must adhere to:
116
+
117
+
<schema>
118
+
{
119
+
"type": "object",
120
+
"properties": {
121
+
"title": {
122
+
"type": "string"
123
+
},
124
+
"ingredients": {
125
+
"type": "array",
126
+
"items": {
127
+
"type": "string"
128
+
}
129
+
},
130
+
"servings": {
131
+
"type": "integer"
132
+
},
133
+
"steps": {
134
+
"type": "array",
135
+
"items": {
136
+
"type": "string"
137
+
}
138
+
},
139
+
"notes": {
140
+
"type": "string"
141
+
}
142
+
},
143
+
"required": ["title", "ingredients", "servings", "steps"]
144
+
}
145
+
</schema>
146
+
```
147
+
</div>
148
+
149
+
This makes the model more likely to spit out objects with the schema you want. You can refine this using llama.cpp BNF from JSON schema, but my runtime of choice ([Ollama](https://ollama.com)) doesn't currently support passing arbitrary grammars at inference time.
150
+
151
+
When I was ripping the video off of TikTok, I accidentally left in a flag I used for YouTube archival that includes the subtitles. I quickly noticed that TikTok subtitles are shockingly good in comparison to YouTube's subtitles. On a lark, I decided to also feed the subtitles into the context window. It helped a lot (especially with videos where the ingredients and amounts were split between the voiceover and the description, presumably in an effort to make this kind of scraping harder because recipes generally aren't considered copyrightable in the US), but I ran into another big problem: consistency.
152
+
153
+
## Few-shot prompting it
154
+
155
+
Sometimes I was getting the right recipe steps and ingredients, other times I was not. Usually the results would end up omitting essential spices or amounts of ingredients. I ended up rethinking what I was doing and then remembered a trick that AI model makers use to make their benchmarks look better: few-shot prompting.
156
+
157
+
<Conv name="Mara" mood="hacker">
158
+
Few shot prompting is when you give the model an example or two of the
159
+
input/output you want and then let the model extrapolate from there. This
160
+
gives the model a better chance of being consistent.
161
+
</Conv>
162
+
163
+
I manually took the subtitles and description of the Kielbasa Sheet Pan meal recipe and reified them into a JSON object manually. Once I fed that into the context window and tested it with a video that hadn't been seen before, I was finally getting consistent results.
164
+
165
+
## The app
166
+
167
+
Now that I was consistently getting what I want, I made [a simple Next.js app](https://github.com/Xe/cooktok) and used [zod](https://zod.dev/) to validate generated objects. When I pass a video URL into the app, it gets processed like this:
168
+
169
+
1. yt-dlp is invoked to download the video, description, thumbnail/poster, and subtitles in English
170
+
2. The description and subtitles are used as input to Hermes 3 with a few-shot pipeline as described above (though I used a chicken orzo recipe instead of the kielbasa sheet pan meal)
171
+
3. The output is parsed as JSON and validated with the zod schema
172
+
4. The result gets rendered to a page so you can print it out and cook the meal
173
+
174
+
This ends up with recipes like this:
175
+
176
+
<Picture
177
+
path="notes/2024/cooktok/recipe-preview"
178
+
desc="Kielbasa Sheet Pan Meal, 4 servings, by @cookinginthemidwest. The asian zing sauce is spicy, so you may want to leave some without it for kids or those who don't like spice, and they can eat theirs with BBQ sauce instead. Ingredients: 2 12 oz packs of Beef Kielbasa, 3 large potatoes, 2 green bell peppers"
179
+
/>
180
+
181
+
<center>
182
+
[Recipe
183
+
PDF](https://cdn.xeiaso.net/file/christine-static/notes/2024/cooktok/Kielbasa-Sheet-Pan-Meal.pdf)
184
+
</center>
185
+
186
+
## Testing
187
+
188
+
Testing this was the hard part due to a prior incident with AI summarization of cooking videos making us make something inedible trying to copy an [Adam Ragusea video on Orange Chicken](https://www.youtube.com/watch?v=zc63e7ZKUrg). I ended up using some deception in order to hide the fact that the Beef Kielbasa Sheet Pan Meal recipe was created using an AI pipeline.
189
+
190
+
Either way, we got all the ingredients and cooked it without consulting the source video:
191
+
192
+
<Picture
193
+
path="notes/2024/cooktok/ingredients"
194
+
desc="A pair of cutting boards with bell peppers, potatoes, one healthy red onion, and a bunch of kielbasa sausage"
195
+
/>
196
+
197
+
<Picture
198
+
path="notes/2024/cooktok/ready-for-oven"
199
+
desc="A sheet pan coated with aluminum foil topped with potatoes, veggies, and sausage"
200
+
/>
201
+
202
+
And then it came out great:
203
+
204
+
<Picture
205
+
path="notes/2024/cooktok/cooked"
206
+
desc="A plate of the resulting creation featuring sausage chunks, potatoes, onions, and red bell pepper"
207
+
/>
208
+
209
+
## Conclusion
210
+
211
+
This is a somewhat viable way to get the recipes out of TikTok cooking videos, although it has a few fundamental limitations that can't easily be worked around yet:
212
+
213
+
1. Many cooking videos on TikTok have their videos function as advertisements for their Patreon, where they give out the full recipes. CookTok cannot work around this, it can only get information from the video metadata.
214
+
2. Some TikTokkers hide or obscure the amounts of ingredients via text on the screen. In a future version, I'll try to use a vision model to extract the text from every 5th frame or so. Maybe just working on keyframes would be fine? I'm not sure, more research is required.
215
+
216
+
For a weekend hack though, I'd call this a resounding success. I played around with some fancy tools and managed to get at least one tasty meal as a result. That's more than enough to be a win in my book.