sequoia
A CLI for publishing standard.site documents to ATProto sequoia.pub
standard site lexicon cli publishing
Star 112
Fork 8
Watch

textContent: Should not contain markdown or other formatting. #44

Closed stevedylan.dev
Closed opened by stevedylan.dev

My last post is a talk, so I tried to be fancy and wrap the slides in figure and figcaption elements in the markdown, and threw in a little custom css for it too, and all of that has been squired into the textContent! see: https://pds.ls/at://did:plc:2ngsl5btroik454wzz7vpbzq/site.standard.document/3mn6cg5pz3e2p

the standard.site docs say: textContent: "Plaintext representation of the documents contents. Should not contain markdown or other formatting." https://standard.site/docs/lexicons/document/#optional-properties

So now i feel like i'm a bad standard.siter. Could sequoia do more cleaning of the text before pushing it into the textContent field?

Thanks for sequoia!

stevedylan.dev

Hey @oli.zilla.org.uk! Thanks for opening this; it looks like our markdown utility isn’t catching everything. @heaths.dev what do you think about adding a dependency to handle markdown/html utilities? I think the manual approach I started with probably isn’t going to scale well since it means catching all HTML too.

heaths.dev

I think it's a good idea. Regex is okay but, as noted, there are just so many different variants and quirks a dedicated library would more likely have a battery of tests to catch all/most of them. strip-markdown seems pretty promising - a plugin to a much larger suite that would provide some options should it come down to it. That said, probably should integrate vite for at least tree-shaking if nothing else.

heaths.dev

With the tests in place, it should be fairly straight-forward to test. Not saying the expected text is perfect - seemed okay to me - but at least gives us some good idea.

@oli.zilla.org.uk could you post a fragment of the markdown that wasn't properly stripped? It would make a good test case.

heaths.dev

Actually, I realized I can just simplify the record you pointed to in your PDS. I've got something working using remark and some plugins - almost no test has changed, exception the ones with headings that make sense to have lines separating them - but would you expect <figure>..</figure> to be stripped entirely? If we're stripping code blocks, it seems like we should simply remove figures as well.

heaths.dev
heaths.dev

@oli.zilla.org.uk that PR is merged. I'm working on a separate one but given the free-form issues with HTML elements, I'm not sure how well that work in practice. One idea is to just convert any text content in HTML (hast) to markdown AST (mast) but many of those tags shouldn't be converted e.g., footnotes.

oli.zilla.org.uk

The most egregious part was I had a style tag at the start of one of my posts. I got rid of that and republished my records. In general my hope was to have html stripped, but the text content to remain. Otherwise my post would become completely empty. I also use things like <small>, for visual effect in posts, that i would want to make it to the textContent field. But I would not want the content of style or script tags getting included, so some judgment is needed.

I also wanted flag that the spec note about markdown in textContent seems to be ignored in practice... https://standard.horse is happily rendering my markdown formatting from the textContent field.

I am new to standard.site lexicons, but I think the intention is to put markdown to the content field: https://github.com/mozzius/standard.horse#content-formats

And of note it looks like strip-markdown is what standard.horse is using: https://github.com/mozzius/standard.horse/blob/bde946c6cf6cbd5a03a13f853316a5e622db2aa3/package.json#L41

heaths.dev

I don’t think it’d be practical to keep a list of HTML nodes (in the remark sense) that should retain their content and those that don’t. HTML isn’t what it used to be: elements can be named anything now. We publish three different ones ourselves.

It may be possible - if exposed (I’ll have to look more closely at the remark plugins) - to handle the basics like <small> but it devolves quickly into the same mess an array of regex did: its an arms race and no one ever wins one of those.

heaths.dev
oli.zilla.org.uk

ok! Could it default to retaining the content always? Putting css in my markdown was a quick hack and gross and i regret ever mentioning it. The rest of my html-in-md is just formatting flair that I would be fine to strip the tags from, but some content is lost if we strip the textContent

Failing that, a "for weirdos like me" --keep-html-text-content flag?

oli.zilla.org.uk

oh it looks like that's what the PR does! nice!

heaths.dev

I don’t know that it’d keep inline <style>. Try out a dev build or define them in your site CSS. The latter is what I do anyway to keep styles consistent.

But let’s open a separate issue about style. Evergreen issues are difficult to track.

heaths.dev

Will be resolved by #82 and #84. We can tweak the HTML conversion in separate issues, pull requests.

Sign up or Sign in to add to the discussion.