Select the types of activity you want to include in your feed.
A simple tool which lets you scrape twitter accounts and crosspost them to bluesky accounts! Comes with a CLI and a webapp for managing profiles! Works with images/videos/link embeds/threads.
···11-# Auto detect text files and perform LF normalization
22-* text=auto
33-44-#
55-# The above will handle all files NOT found below
66-#
77-88-# Code
99-*.js text eol=lf
1010-*.ts text eol=lf
1111-1212-# Documents
1313-*.bibtex text diff=bibtex
1414-*.doc diff=astextplain
1515-*.DOC diff=astextplain
1616-*.docx diff=astextplain
1717-*.DOCX diff=astextplain
1818-*.dot diff=astextplain
1919-*.DOT diff=astextplain
2020-*.pdf diff=astextplain
2121-*.PDF diff=astextplain
2222-*.rtf diff=astextplain
2323-*.RTF diff=astextplain
2424-*.md text diff=markdown
2525-*.mdx text diff=markdown
2626-*.tex text diff=tex
2727-*.adoc text
2828-*.textile text
2929-*.mustache text
3030-*.csv text
3131-*.tab text
3232-*.tsv text
3333-*.txt text
3434-*.sql text
3535-3636-# Graphics
3737-*.png binary
3838-*.jpg binary
3939-*.jpeg binary
4040-*.gif binary
4141-*.tif binary
4242-*.tiff binary
4343-*.ico binary
4444-*.svg text
4545-*.eps binary
4646-4747-# Scripts
4848-*.bash text eol=lf
4949-*.fish text eol=lf
5050-*.sh text eol=lf
5151-*.zsh text eol=lf
5252-*.bat text eol=crlf
5353-*.cmd text eol=crlf
5454-*.ps1 text eol=crlf
5555-5656-# Serialisation
5757-*.json text
5858-*.toml text
5959-*.xml text
6060-*.yaml text
6161-*.yml text
6262-6363-# Archives
6464-*.7z binary
6565-*.gz binary
6666-*.tar binary
6767-*.tgz binary
6868-*.zip binary
6969-7070-# Text files where line endings should be preserved
7171-*.patch -text
7272-7373-#
7474-# Exclude files from exporting
7575-#
7676-7777-.gitattributes export-ignore
7878-.gitignore export-ignore
···11-MIT License
22-33-Copyright (c) 2022 karashiiro
44-55-Permission is hereby granted, free of charge, to any person obtaining a copy
66-of this software and associated documentation files (the "Software"), to deal
77-in the Software without restriction, including without limitation the rights
88-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
99-copies of the Software, and to permit persons to whom the Software is
1010-furnished to do so, subject to the following conditions:
1111-1212-The above copyright notice and this permission notice shall be included in all
1313-copies or substantial portions of the Software.
1414-1515-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
1616-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
1717-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
1818-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
1919-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
2020-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
2121-SOFTWARE.
-295
twitter-scraper/README.md
···11-# twitter-scraper
22-33-[](https://the-convocation.github.io/twitter-scraper/)
44-55-A port of the now-archived [n0madic/twitter-scraper](https://github.com/n0madic/twitter-scraper) to Node.js.
66-77-> Twitter's API is annoying to work with, and has lots of limitations — luckily
88-> their frontend (JavaScript) has it's own API, which I reverse-engineered. No
99-> API rate limits. No tokens needed. No restrictions. Extremely fast.
1010->
1111-> You can use this library to get the text of any user's Tweets trivially.
1212-1313-Many things have changed since X (the company formerly known as Twitter) was acquired in 2022:
1414-1515-- Several operations require logging in with a real user account via
1616- `scraper.login()`. **While we are not aware of confirmed cases caused
1717- by this library, any account you log into with this library is subject
1818- to being banned at any time. You have been warned.**
1919-- Twitter's frontend API does in fact have rate limits
2020- ([#11](https://github.com/the-convocation/twitter-scraper/issues/11)).
2121- The rate limits are dynamic and sometimes change, so we don't know
2222- exactly what they are at all times. Refer to [rate limiting](#rate-limiting)
2323- for more information.
2424-- Twitter's authentication requirements and frontend API endpoints
2525- change frequently, breaking this library. Fixes for these issues
2626- typically take at least a few days to go out.
2727-2828-## Installation
2929-3030-This package requires Node.js v16.0.0 or greater.
3131-3232-NPM:
3333-3434-```sh
3535-npm install @the-convocation/twitter-scraper
3636-```
3737-3838-Yarn:
3939-4040-```sh
4141-yarn add @the-convocation/twitter-scraper
4242-```
4343-4444-TypeScript types have been bundled with the distribution.
4545-4646-## Usage
4747-4848-Most use cases are exactly the same as in
4949-[n0madic/twitter-scraper](https://github.com/n0madic/twitter-scraper). Channel
5050-iterators have been translated into
5151-[AsyncGenerator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/AsyncGenerator)
5252-instances, and can be consumed with the corresponding
5353-`for await (const x of y) { ... }` syntax.
5454-5555-### Browser usage
5656-5757-This package directly invokes the Twitter API, which does not have permissive
5858-CORS headers. With the default settings, requests will fail unless you disable
5959-CORS checks, which is not advised. Instead, applications must provide a CORS
6060-proxy and configure it in the `Scraper` options.
6161-6262-Proxies (and other request mutations) can be configured with the request
6363-interceptor transform:
6464-6565-```ts
6666-const scraper = new Scraper({
6767- transform: {
6868- request(input: RequestInfo | URL, init?: RequestInit) {
6969- // The arguments here are the same as the parameters to fetch(), and
7070- // are kept as-is for flexibility of both the library and applications.
7171- if (input instanceof URL) {
7272- const proxy =
7373- 'https://corsproxy.io/?' + encodeURIComponent(input.toString());
7474- return [proxy, init];
7575- } else if (typeof input === 'string') {
7676- const proxy = 'https://corsproxy.io/?' + encodeURIComponent(input);
7777- return [proxy, init];
7878- } else {
7979- // Omitting handling for example
8080- throw new Error('Unexpected request input type');
8181- }
8282- },
8383- },
8484-});
8585-```
8686-8787-[corsproxy.io](https://corsproxy.io) is a public CORS proxy that works correctly
8888-with this package.
8989-9090-The public CORS proxy [corsproxy.org](https://corsproxy.org) _does not work_ at
9191-the time of writing (at least not using their recommended integration on the
9292-front page).
9393-9494-#### Next.js 13.x example:
9595-9696-```tsx
9797-'use client';
9898-9999-import { Scraper, Tweet } from '@the-convocation/twitter-scraper';
100100-import { useEffect, useMemo, useState } from 'react';
101101-102102-export default function Home() {
103103- const scraper = useMemo(
104104- () =>
105105- new Scraper({
106106- transform: {
107107- request(input: RequestInfo | URL, init?: RequestInit) {
108108- if (input instanceof URL) {
109109- const proxy =
110110- 'https://corsproxy.io/?' + encodeURIComponent(input.toString());
111111- return [proxy, init];
112112- } else if (typeof input === 'string') {
113113- const proxy =
114114- 'https://corsproxy.io/?' + encodeURIComponent(input);
115115- return [proxy, init];
116116- } else {
117117- throw new Error('Unexpected request input type');
118118- }
119119- },
120120- },
121121- }),
122122- [],
123123- );
124124- const [tweet, setTweet] = useState<Tweet | null>(null);
125125-126126- useEffect(() => {
127127- async function getTweet() {
128128- const latestTweet = await scraper.getLatestTweet('twitter');
129129- if (latestTweet) {
130130- setTweet(latestTweet);
131131- }
132132- }
133133-134134- getTweet();
135135- }, [scraper]);
136136-137137- return (
138138- <main className="flex min-h-screen flex-col items-center justify-between p-24">
139139- {tweet?.text}
140140- </main>
141141- );
142142-}
143143-```
144144-145145-### Edge runtimes
146146-147147-This package currently uses
148148-[`cross-fetch`](https://www.npmjs.com/package/cross-fetch) as a portable
149149-`fetch`. Edge runtimes such as CloudFlare Workers sometimes have `fetch`
150150-functions that behave differently from the web standard, so you may need to
151151-override the `fetch` function the scraper uses. If so, a custom `fetch` can be
152152-provided in the options:
153153-154154-```ts
155155-const scraper = new Scraper({
156156- fetch: fetch,
157157-});
158158-```
159159-160160-Note that this does not change the arguments passed to the function, or the
161161-expected return type. If the custom `fetch` function produces runtime errors
162162-related to incorrect types, be sure to wrap it in a shim (not currently
163163-supported directly by interceptors):
164164-165165-```ts
166166-const scraper = new Scraper({
167167- fetch: (input, init) => {
168168- // Transform input and init into your function's expected types...
169169- return fetch(input, init).then((res) => {
170170- // Transform res into a web-compliant response...
171171- return res;
172172- });
173173- },
174174-});
175175-```
176176-177177-### Bypassing Cloudflare bot detection
178178-179179-In some cases, Twitter's authentication endpoints may be protected by Cloudflare's advanced bot detection, resulting in `403 Forbidden` errors during login. This typically happens because standard Node.js TLS fingerprints are detected as non-browser clients.
180180-181181-To bypass this protection, you can use the optional CycleTLS `fetch` integration to mimic Chrome browser TLS fingerprints:
182182-183183-**Installation:**
184184-185185-```sh
186186-npm install cycletls
187187-# or
188188-yarn add cycletls
189189-```
190190-191191-**Usage:**
192192-193193-```ts
194194-import { Scraper } from '@the-convocation/twitter-scraper';
195195-import {
196196- cycleTLSFetch,
197197- cycleTLSExit,
198198-} from '@the-convocation/twitter-scraper/cycletls';
199199-200200-const scraper = new Scraper({
201201- fetch: cycleTLSFetch,
202202-});
203203-204204-// Use the scraper normally
205205-await scraper.login(username, password, email);
206206-207207-// Important: cleanup CycleTLS resources when done
208208-cycleTLSExit();
209209-```
210210-211211-**Note:** The `/cycletls` entrypoint is Node.js only and will not work in browser environments. It's provided as a separate optional entrypoint to avoid bundling binaries in environments where they cannot run.
212212-213213-See the [cycletls example](./examples/cycletls/) for a complete working example.
214214-215215-### Rate limiting
216216-217217-The Twitter API heavily rate-limits clients, requiring that the scraper has its own
218218-rate-limit handling to behave predictably when rate-limiting occurs. By default, the
219219-scraper uses a rate-limiting strategy that waits for the current rate-limiting period
220220-to expire before resuming requests.
221221-222222-**This has been known to take a very long time, in some cases (up to 13 minutes).**
223223-224224-You may want to change how rate-limiting events are handled, potentially by pooling
225225-scrapers logged-in to different accounts (refer to [#116](https://github.com/the-convocation/twitter-scraper/pull/116) for how to do this yourself). The rate-limit handling strategy can be configured by passing a custom
226226-implementation to the `rateLimitStrategy` option in the scraper constructor:
227227-228228-```ts
229229-import { Scraper, RateLimitStrategy } from '@the-convocation/twitter-scraper';
230230-231231-class CustomRateLimitStrategy implements RateLimitStrategy {
232232- async onRateLimit(event: RateLimitEvent): Promise<void> {
233233- // your own logic...
234234- }
235235-}
236236-237237-const scraper = new Scraper({
238238- rateLimitStrategy: new CustomRateLimitStrategy(),
239239-});
240240-```
241241-242242-More information on this interface can be found on the [`RateLimitStrategy`](https://the-convocation.github.io/twitter-scraper/interfaces/RateLimitStrategy.html)
243243-page in the documentation. The library provides two pre-written implementations to choose from:
244244-245245-- `WaitingRateLimitStrategy`: The default, which waits for the limit to expire.
246246-- `ErrorRateLimitStrategy`: A strategy that throws if any rate-limit event occurs.
247247-248248-## Contributing
249249-250250-### Setup
251251-252252-This project currently requires Node 18.x for development and uses Yarn for
253253-package management.
254254-[Corepack](https://nodejs.org/dist/latest-v18.x/docs/api/corepack.html) is
255255-configured for this project, so you don't need to install a particular package
256256-manager version manually.
257257-258258-> The project supports Node 16.x at runtime, but requires Node 18.x to run its
259259-> build tools.
260260-261261-Just run `corepack enable` to turn on the shims, then run `yarn` to install the
262262-dependencies.
263263-264264-#### Basic scripts
265265-266266-- `yarn build`: Builds the project into the `dist` folder
267267-- `yarn test`: Runs the package tests (see [Testing](#testing) first)
268268-269269-Run `yarn help` for general `yarn` usage information.
270270-271271-### Testing
272272-273273-This package includes unit tests for all major functionality. Given the speed at
274274-which Twitter's private API changes, failing tests are to be expected.
275275-276276-```sh
277277-yarn test
278278-```
279279-280280-Before running tests, you should configure environment variables for
281281-authentication.
282282-283283-```
284284-TWITTER_USERNAME= # Account username
285285-TWITTER_PASSWORD= # Account password
286286-TWITTER_EMAIL= # Account email
287287-TWITTER_COOKIES= # JSON-serialized array of cookies of an authenticated session
288288-PROXY_URL= # HTTP(s) proxy for requests (optional)
289289-```
290290-291291-### Commit message format
292292-293293-We use [Conventional Commits](https://www.conventionalcommits.org), and enforce
294294-this with precommit checks. Please refer to the Git history for real examples of
295295-the commit message format.
···11-# CycleTLS Cloudflare Bypass Example
22-33-This example demonstrates how to use the `@the-convocation/twitter-scraper/cycletls` entrypoint to bypass Cloudflare bot detection when authenticating with Twitter.
44-55-## Problem
66-77-Twitter's authentication endpoints may be protected by Cloudflare's bot detection, which analyzes TLS fingerprints to detect non-browser clients. Standard Node.js TLS handshakes can trigger `403 Forbidden` errors during login.
88-99-## Solution
1010-1111-This example uses [CycleTLS](https://github.com/Danny-Dasilva/CycleTLS) to mimic Chrome browser TLS fingerprints, allowing requests to pass through Cloudflare's protection.
1212-1313-## Installation
1414-1515-```sh
1616-yarn install
1717-```
1818-1919-## Configuration
2020-2121-Create a `.env` file in this directory with your Twitter credentials:
2222-2323-```
2424-TWITTER_USERNAME=your_username
2525-TWITTER_PASSWORD=your_password
2626-TWITTER_EMAIL=your_email
2727-```
2828-2929-## Usage
3030-3131-```sh
3232-yarn start
3333-```
3434-3535-## How it works
3636-3737-The example imports the `cycleTLSFetch` function from the `/cycletls` subpath:
3838-3939-```ts
4040-import { Scraper } from '@the-convocation/twitter-scraper';
4141-import { cycleTLSFetch, cycleTLSExit } from '@the-convocation/twitter-scraper/cycletls';
4242-4343-const scraper = new Scraper({
4444- fetch: cycleTLSFetch,
4545-});
4646-```
4747-4848-This replaces the default fetch implementation with one that uses Chrome-like TLS fingerprints, bypassing Cloudflare's detection.
···11-# Copy this file to .env.local and update these values if needed.
22-VITE_TWITTER_USERNAME=$TWITTER_USERNAME
33-VITE_TWITTER_PASSWORD=$TWITTER_PASSWORD
44-VITE_TWITTER_EMAIL=$TWITTER_EMAIL
···11-# React Example
22-33-Browser usage example in React. Due to Twitter's CORS headers not allowing external websites from calling their APIs,
44-this requires using a CORS proxy of some kind.
55-66-## Running
77-88-First, copy `.env.example` to a new `.env.local` file, and update the environment variables to point to your
99-own account credentials if needed.
1010-1111-In the `cors-proxy` example folder, run the following command:
1212-1313-```bash
1414-yarn start
1515-```
1616-1717-Then, in this folder, run the following command to start the Vite development server:
1818-1919-```bash
2020-yarn dev
2121-```
···11-/**
22- * CycleTLS fetch wrapper for bypassing Cloudflare bot detection
33- *
44- * This is a separate entrypoint to avoid bundling cycletls in environments
55- * where it cannot run (like browsers). Import from '@the-convocation/twitter-scraper/cycletls'
66- */
77-export { cycleTLSFetch, cycleTLSExit } from './cycletls-fetch';
-58
twitter-scraper/src/_module.ts
···11-export type { FetchTransformOptions } from './api';
22-export type { FetchParameters } from './api-types';
33-export type {
44- TwitterUserAuthCredentials,
55- TwitterUserAuthFlowInitRequest,
66- TwitterUserAuthFlowSubtaskRequest,
77- TwitterUserAuthFlowRequest,
88- TwitterUserAuthFlowResponse,
99- FlowSubtaskHandler,
1010- FlowSubtaskHandlerApi,
1111- FlowTokenResult,
1212- FlowTokenResultError,
1313- FlowTokenResultSuccess,
1414-} from './auth-user';
1515-export type {
1616- DmInboxResponse,
1717- DmInbox,
1818- DmConversationResponse,
1919- DmConversationTimeline,
2020- DmConversation,
2121- DmStatus,
2222- DmParticipant,
2323- DmMessageEntry,
2424- DmMessage,
2525- DmMessageData,
2626- DmReaction,
2727- DmMessageEntities,
2828- DmMessageUrl,
2929- DmWelcomeMessage,
3030- DmInboxTimelines,
3131- DmTimelineState,
3232-} from './direct-messages';
3333-export {
3434- ApiError,
3535- AuthenticationError,
3636- type TwitterApiErrorRaw,
3737- type TwitterApiErrorExtensions,
3838- type TwitterApiErrorPosition,
3939- type TwitterApiErrorTraceInfo,
4040-} from './errors';
4141-export type { Profile } from './profile';
4242-export {
4343- type RateLimitEvent,
4444- type RateLimitStrategy,
4545- WaitingRateLimitStrategy,
4646- ErrorRateLimitStrategy,
4747-} from './rate-limit';
4848-export { Scraper, type ScraperOptions } from './scraper';
4949-export { SearchMode } from './search';
5050-export type { QueryProfilesResponse, QueryTweetsResponse } from './timeline-v1';
5151-export type {
5252- Tweet,
5353- TweetQuery,
5454- Mention,
5555- Photo,
5656- PlaceRaw,
5757- Video,
5858-} from './tweets';
-179
twitter-scraper/src/api-data.ts
···11-import stringify from 'json-stable-stringify';
22-33-/**
44- * Examples of requests to API endpoints. These are parsed at runtime and used
55- * as templates for requests to a particular endpoint. Please ensure these do
66- * not contain any information that you do not want published to NPM.
77- */
88-const endpoints = {
99- UserTweets:
1010- 'https://x.com/i/api/graphql/oRJs8SLCRNRbQzuZG93_oA/UserTweets?variables=%7B%22userId%22%3A%221806359170830172162%22%2C%22count%22%3A20%2C%22includePromotedContent%22%3Atrue%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22withVoice%22%3Atrue%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22payments_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22responsive_web_profile_redirect_enabled%22%3Afalse%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Atrue%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D&fieldToggles=%7B%22withArticlePlainText%22%3Afalse%7D',
1111- UserTweetsAndReplies:
1212- 'https://x.com/i/api/graphql/Hk4KlJ-ONjlJsucqR55P7g/UserTweetsAndReplies?variables=%7B%22userId%22%3A%221806359170830172162%22%2C%22count%22%3A20%2C%22includePromotedContent%22%3Atrue%2C%22withCommunity%22%3Atrue%2C%22withVoice%22%3Atrue%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Afalse%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Afalse%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D&fieldToggles=%7B%22withArticlePlainText%22%3Afalse%7D',
1313- UserLikedTweets:
1414- 'https://x.com/i/api/graphql/XHTMjDbiTGLQ9cP1em-aqQ/Likes?variables=%7B%22userId%22%3A%222244196397%22%2C%22count%22%3A20%2C%22includePromotedContent%22%3Afalse%2C%22withClientEventToken%22%3Afalse%2C%22withBirdwatchNotes%22%3Afalse%2C%22withVoice%22%3Atrue%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Afalse%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Afalse%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D&fieldToggles=%7B%22withArticlePlainText%22%3Afalse%7D',
1515- UserByScreenName:
1616- 'https://api.x.com/graphql/-oaLodhGbbnzJBACb1kk2Q/UserByScreenName?variables=%7B%22screen_name%22%3A%22geminiapp%22%2C%22withGrokTranslatedBio%22%3Afalse%7D&features=%7B%22hidden_profile_subscriptions_enabled%22%3Atrue%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22responsive_web_profile_redirect_enabled%22%3Afalse%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22subscriptions_verification_info_is_identity_verified_enabled%22%3Atrue%2C%22subscriptions_verification_info_verified_since_enabled%22%3Atrue%2C%22highlights_tweets_tab_ui_enabled%22%3Atrue%2C%22responsive_web_twitter_article_notes_tab_enabled%22%3Atrue%2C%22subscriptions_feature_can_gift_premium%22%3Atrue%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%7D&fieldToggles=%7B%22withPayments%22%3Afalse%2C%22withAuxiliaryUserLabels%22%3Atrue%7D',
1717- TweetDetail:
1818- 'https://x.com/i/api/graphql/YVyS4SfwYW7Uw5qwy0mQCA/TweetDetail?variables=%7B%22focalTweetId%22%3A%221985465713096794294%22%2C%22referrer%22%3A%22profile%22%2C%22with_rux_injections%22%3Afalse%2C%22rankingMode%22%3A%22Relevance%22%2C%22includePromotedContent%22%3Atrue%2C%22withCommunity%22%3Atrue%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22withBirdwatchNotes%22%3Atrue%2C%22withVoice%22%3Atrue%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22payments_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22responsive_web_profile_redirect_enabled%22%3Afalse%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Atrue%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D&fieldToggles=%7B%22withArticleRichContentState%22%3Atrue%2C%22withArticlePlainText%22%3Afalse%2C%22withGrokAnalyze%22%3Afalse%2C%22withDisallowedReplyControls%22%3Afalse%7D',
1919- TweetResultByRestId:
2020- 'https://api.x.com/graphql/tCVRZ3WCvoj0BVO7BKnL-Q/TweetResultByRestId?variables=%7B%22tweetId%22%3A%221985465713096794294%22%2C%22withCommunity%22%3Afalse%2C%22includePromotedContent%22%3Afalse%2C%22withVoice%22%3Afalse%7D&features=%7B%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Afalse%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Afalse%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22payments_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22responsive_web_profile_redirect_enabled%22%3Afalse%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D&fieldToggles=%7B%22withArticleRichContentState%22%3Atrue%2C%22withArticlePlainText%22%3Afalse%2C%22withGrokAnalyze%22%3Afalse%2C%22withDisallowedReplyControls%22%3Afalse%7D',
2121- ListTweets:
2222- 'https://x.com/i/api/graphql/S1Sm3_mNJwa-fnY9htcaAQ/ListLatestTweetsTimeline?variables=%7B%22listId%22%3A%221736495155002106192%22%2C%22count%22%3A20%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Afalse%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Afalse%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D',
2323- SearchTimeline:
2424- 'https://x.com/i/api/graphql/nK1dw4oV3k4w5TdtcAdSww/SearchTimeline?variables=%7B%22rawQuery%22%3A%22twitter%22%2C%22count%22%3A20%2C%22querySource%22%3A%22typed_query%22%2C%22product%22%3A%22Top%22%2C%22withGrokTranslatedBio%22%3Afalse%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22responsive_web_profile_redirect_enabled%22%3Afalse%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Atrue%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22tweetypie_unmention_optimization_enabled%22%3Atrue%2C%22responsive_web_media_download_video_enabled%22%3Afalse%2C%22rweb_lists_timeline_redesign_enabled%22%3Atrue%7D',
2525- Followers:
2626- 'https://x.com/i/api/graphql/SCu9fVIlCUm-BM8-tL5pkQ/Followers?variables=%7B%22userId%22%3A%221806359170830172162%22%2C%22count%22%3A20%2C%22includePromotedContent%22%3Afalse%2C%22withGrokTranslatedBio%22%3Afalse%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22responsive_web_profile_redirect_enabled%22%3Afalse%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Atrue%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D',
2727- Following:
2828- 'https://x.com/i/api/graphql/S5xUN9s2v4xk50KWGGvyvQ/Following?variables=%7B%22userId%22%3A%221806359170830172162%22%2C%22count%22%3A20%2C%22includePromotedContent%22%3Afalse%2C%22withGrokTranslatedBio%22%3Afalse%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22responsive_web_profile_redirect_enabled%22%3Afalse%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Atrue%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D',
2929-} as const;
3030-3131-export interface EndpointFieldInfo {
3232- /**
3333- * Request variables, used for providing arguments such as user IDs or result counts.
3434- */
3535- variables: Record<string, unknown>;
3636-3737- /**
3838- * Request features, used for encoding feature flags into the request. These may either be
3939- * boolean values or numerically-encoded booleans (1 or 0). It is possible this may change
4040- * to include other representations of booleans as Twitter's backend evolves.
4141- */
4242- features: Record<string, unknown>;
4343-4444- /**
4545- * Request field toggles, used for limiting how returned fields are represented. This is
4646- * rarely used.
4747- */
4848- fieldToggles: Record<string, unknown>;
4949-}
5050-5151-type SomePartial<T, K extends keyof T> = Omit<T, K> & Partial<Pick<T, K>>;
5252-5353-type EndpointVersion = string;
5454-type EndpointName = string;
5555-type EncodedVariables = string;
5656-type EncodedFeatures = string;
5757-type EncodedFieldToggles = string;
5858-5959-// TODO: Set up field-level Intellisense for the QraphQL parameters in these?
6060-type BaseUrl =
6161- | 'https://twitter.com/i/api/graphql'
6262- | 'https://x.com/i/api/graphql'
6363- | 'https://api.x.com/graphql';
6464-type EndpointFields<EndpointUrl> =
6565- EndpointUrl extends `${BaseUrl}/${EndpointVersion}/${EndpointName}?variables=${EncodedVariables}&features=${EncodedFeatures}&fieldToggles=${EncodedFieldToggles}`
6666- ? EndpointFieldInfo
6767- : EndpointUrl extends `${BaseUrl}/${EndpointVersion}/${EndpointName}?variables=${EncodedVariables}&features=${EncodedFeatures}`
6868- ? SomePartial<EndpointFieldInfo, 'fieldToggles'>
6969- : EndpointUrl extends `${BaseUrl}/${EndpointVersion}/${EndpointName}?variables=${EncodedVariables}`
7070- ? SomePartial<EndpointFieldInfo, 'features' | 'fieldToggles'>
7171- : Partial<EndpointFieldInfo>;
7272-7373-export type ApiRequestInfo<EndpointUrl> = EndpointFields<EndpointUrl> & {
7474- /**
7575- * The URL, without any GraphQL query parameters.
7676- */
7777- url: string;
7878-7979- /**
8080- * Converts the request back into a URL to be sent to the Twitter API.
8181- */
8282- toRequestUrl(): string;
8383-};
8484-8585-/** Wrapper class for API request information. */
8686-class ApiRequest<EndpointUrl> {
8787- url: string;
8888- variables?: Record<string, unknown> | undefined;
8989- features?: Record<string, unknown> | undefined;
9090- fieldToggles?: Record<string, unknown> | undefined;
9191-9292- constructor(info: Omit<ApiRequestInfo<EndpointUrl>, 'toRequestUrl'>) {
9393- this.url = info.url;
9494- this.variables = info.variables;
9595- this.features = info.features;
9696- this.fieldToggles = info.fieldToggles;
9797- }
9898-9999- toRequestUrl(): string {
100100- const params = new URLSearchParams();
101101-102102- // Only include query parameters with values
103103- if (this.variables) {
104104- // Stringify with the query keys in sorted order like the Go package
105105- const variablesStr = stringify(this.variables);
106106- if (variablesStr) params.set('variables', variablesStr);
107107- }
108108-109109- if (this.features) {
110110- const featuresStr = stringify(this.features);
111111- if (featuresStr) params.set('features', featuresStr);
112112- }
113113-114114- if (this.fieldToggles) {
115115- const fieldTogglesStr = stringify(this.fieldToggles);
116116- if (fieldTogglesStr) params.set('fieldToggles', fieldTogglesStr);
117117- }
118118-119119- return `${this.url}?${params.toString()}`;
120120- }
121121-}
122122-123123-/**
124124- * Parses information from a Twitter API endpoint using an example request
125125- * URL against that endpoint. This can be used to extract GraphQL parameters
126126- * in order to easily reuse and/or override them later.
127127- * @param example An example of the endpoint to analyze.
128128- * @returns The parsed endpoint information.
129129- */
130130-function parseEndpointExample<
131131- Endpoints,
132132- Endpoint extends string & keyof Endpoints,
133133->(example: Endpoint): ApiRequestInfo<Endpoints[Endpoint]> {
134134- const { protocol, host, pathname, searchParams: query } = new URL(example);
135135-136136- const base = `${protocol}//${host}${pathname}`;
137137- const variables = query.get('variables');
138138- const features = query.get('features');
139139- const fieldToggles = query.get('fieldToggles');
140140-141141- return new ApiRequest<Endpoints[Endpoint]>({
142142- url: base,
143143- variables: variables ? JSON.parse(variables) : undefined,
144144- features: features ? JSON.parse(features) : undefined,
145145- fieldToggles: fieldToggles ? JSON.parse(fieldToggles) : undefined,
146146- } as Omit<
147147- ApiRequestInfo<Endpoints[Endpoint]>,
148148- 'toRequestUrl'
149149- >) as ApiRequestInfo<Endpoints[Endpoint]>;
150150-}
151151-152152-type ApiRequestFactory<Endpoints> = {
153153- [Endpoint in keyof Endpoints as `create${string &
154154- Endpoint}Request`]: () => ApiRequestInfo<Endpoints[Endpoint]>;
155155-};
156156-157157-function createApiRequestFactory<Endpoints extends Record<string, string>>(
158158- endpoints: Endpoints,
159159-): ApiRequestFactory<Endpoints> {
160160- type UntypedApiRequestFactory = ApiRequestFactory<Record<string, string>>;
161161-162162- return Object.entries(endpoints)
163163- .map<UntypedApiRequestFactory>(([endpointName, endpointExample]) => {
164164- // Create a partial factory for only one endpoint
165165- return {
166166- [`create${endpointName}Request`]: () => {
167167- // Create a new instance on each invocation so that we can safely
168168- // mutate requests before sending them off
169169- return parseEndpointExample<Endpoints, any>(endpointExample);
170170- },
171171- };
172172- })
173173- .reduce((agg, next) => {
174174- // Merge all of our factories into one that includes every endpoint
175175- return Object.assign(agg, next);
176176- }) as ApiRequestFactory<Endpoints>;
177177-}
178178-179179-export const apiRequestFactory = createApiRequestFactory(endpoints);
-3
twitter-scraper/src/api-types.ts
···11-// For some reason using Parameters<typeof fetch> reduces the request transform function to
22-// `(url: string) => string` in tests.
33-export type FetchParameters = [input: RequestInfo | URL, init?: RequestInit];
-222
twitter-scraper/src/api.ts
···11-import { FetchParameters } from './api-types';
22-import { TwitterAuth, TwitterGuestAuth } from './auth';
33-import { ApiError } from './errors';
44-import { Platform, PlatformExtensions } from './platform';
55-import { updateCookieJar } from './requests';
66-import { Headers } from 'headers-polyfill';
77-import debug from 'debug';
88-import { generateTransactionId } from './xctxid';
99-1010-const log = debug('twitter-scraper:api');
1111-1212-export interface FetchTransformOptions {
1313- /**
1414- * Transforms the request options before a request is made. This executes after all of the default
1515- * parameters have been configured, and is stateless. It is safe to return new request options
1616- * objects.
1717- * @param args The request options.
1818- * @returns The transformed request options.
1919- */
2020- request: (
2121- ...args: FetchParameters
2222- ) => FetchParameters | Promise<FetchParameters>;
2323-2424- /**
2525- * Transforms the response after a request completes. This executes immediately after the request
2626- * completes, and is stateless. It is safe to return a new response object.
2727- * @param response The response object.
2828- * @returns The transformed response object.
2929- */
3030- response: (response: Response) => Response | Promise<Response>;
3131-}
3232-3333-export const bearerToken =
3434- 'AAAAAAAAAAAAAAAAAAAAAFQODgEAAAAAVHTp76lzh3rFzcHbmHVvQxYYpTw%3DckAlMINMjmCwxUcaXbAN4XqJVdgMJaHqNOFgPMK0zN1qLqLQCF';
3535-3636-export const bearerToken2 =
3737- 'AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA';
3838-3939-export async function jitter(maxMs: number): Promise<void> {
4040- const jitter = Math.random() * maxMs;
4141- await new Promise((resolve) => setTimeout(resolve, jitter));
4242-}
4343-4444-/**
4545- * An API result container.
4646- */
4747-export type RequestApiResult<T> =
4848- | { success: true; value: T }
4949- | { success: false; err: Error };
5050-5151-/**
5252- * Used internally to send HTTP requests to the Twitter API.
5353- * @internal
5454- * @param url - The URL to send the request to.
5555- * @param auth - The instance of {@link TwitterAuth} that will be used to authorize this request.
5656- * @param method - The HTTP method used when sending this request.
5757- * @param platform - The platform extensions to use.
5858- * @param headers - The headers to include in the request.
5959- * @param bearerTokenOverride - Optional bearer token to use instead of the default one.
6060- */
6161-export async function requestApi<T>(
6262- url: string,
6363- auth: TwitterAuth,
6464- method: 'GET' | 'POST' = 'GET',
6565- platform: PlatformExtensions = new Platform(),
6666- headers: Headers = new Headers(),
6767- bearerTokenOverride?: string,
6868-): Promise<RequestApiResult<T>> {
6969- log(`Making ${method} request to ${url}`);
7070-7171- await auth.installTo(headers, url, bearerTokenOverride);
7272- await platform.randomizeCiphers();
7373-7474- if (
7575- auth instanceof TwitterGuestAuth &&
7676- auth.options?.experimental?.xClientTransactionId
7777- ) {
7878- const transactionId = await generateTransactionId(
7979- url,
8080- auth.fetch.bind(auth),
8181- method,
8282- );
8383- headers.set('x-client-transaction-id', transactionId);
8484- }
8585-8686- let res: Response;
8787- do {
8888- const fetchParameters: FetchParameters = [
8989- url,
9090- {
9191- method,
9292- headers,
9393- credentials: 'include',
9494- },
9595- ];
9696-9797- try {
9898- res = await auth.fetch(...fetchParameters);
9999- } catch (err) {
100100- if (!(err instanceof Error)) {
101101- throw err;
102102- }
103103-104104- return {
105105- success: false,
106106- err: new Error('Failed to perform request.'),
107107- };
108108- }
109109-110110- await updateCookieJar(auth.cookieJar(), res.headers);
111111-112112- if (res.status === 429) {
113113- log('Rate limit hit, waiting for retry...');
114114- await auth.onRateLimit({
115115- fetchParameters: fetchParameters,
116116- response: res,
117117- });
118118- }
119119- } while (res.status === 429);
120120-121121- if (!res.ok) {
122122- return {
123123- success: false,
124124- err: await ApiError.fromResponse(res),
125125- };
126126- }
127127-128128- const value: T = await flexParseJson(res);
129129- if (res.headers.get('x-rate-limit-incoming') == '0') {
130130- auth.deleteToken();
131131- return { success: true, value };
132132- } else {
133133- return { success: true, value };
134134- }
135135-}
136136-137137-export async function flexParseJson<T>(res: Response): Promise<T> {
138138- try {
139139- return await res.json();
140140- } catch {
141141- log('Failed to parse response as JSON, trying text parse...');
142142- const text = await res.text();
143143- log('Response text:', text);
144144- return JSON.parse(text);
145145- }
146146-}
147147-148148-/** @internal */
149149-export function addApiFeatures(o: object) {
150150- return {
151151- ...o,
152152- rweb_lists_timeline_redesign_enabled: true,
153153- responsive_web_graphql_exclude_directive_enabled: true,
154154- verified_phone_label_enabled: false,
155155- creator_subscriptions_tweet_preview_api_enabled: true,
156156- responsive_web_graphql_timeline_navigation_enabled: true,
157157- responsive_web_graphql_skip_user_profile_image_extensions_enabled: false,
158158- tweetypie_unmention_optimization_enabled: true,
159159- responsive_web_edit_tweet_api_enabled: true,
160160- graphql_is_translatable_rweb_tweet_is_translatable_enabled: true,
161161- view_counts_everywhere_api_enabled: true,
162162- longform_notetweets_consumption_enabled: true,
163163- tweet_awards_web_tipping_enabled: false,
164164- freedom_of_speech_not_reach_fetch_enabled: true,
165165- standardized_nudges_misinfo: true,
166166- longform_notetweets_rich_text_read_enabled: true,
167167- responsive_web_enhance_cards_enabled: false,
168168- subscriptions_verification_info_enabled: true,
169169- subscriptions_verification_info_reason_enabled: true,
170170- subscriptions_verification_info_verified_since_enabled: true,
171171- super_follow_badge_privacy_enabled: false,
172172- super_follow_exclusive_tweet_notifications_enabled: false,
173173- super_follow_tweet_api_enabled: false,
174174- super_follow_user_api_enabled: false,
175175- android_graphql_skip_api_media_color_palette: false,
176176- creator_subscriptions_subscription_count_enabled: false,
177177- blue_business_profile_image_shape_enabled: false,
178178- unified_cards_ad_metadata_container_dynamic_card_content_query_enabled:
179179- false,
180180- };
181181-}
182182-183183-export function addApiParams(
184184- params: URLSearchParams,
185185- includeTweetReplies: boolean,
186186-): URLSearchParams {
187187- params.set('include_profile_interstitial_type', '1');
188188- params.set('include_blocking', '1');
189189- params.set('include_blocked_by', '1');
190190- params.set('include_followed_by', '1');
191191- params.set('include_want_retweets', '1');
192192- params.set('include_mute_edge', '1');
193193- params.set('include_can_dm', '1');
194194- params.set('include_can_media_tag', '1');
195195- params.set('include_ext_has_nft_avatar', '1');
196196- params.set('include_ext_is_blue_verified', '1');
197197- params.set('include_ext_verified_type', '1');
198198- params.set('skip_status', '1');
199199- params.set('cards_platform', 'Web-12');
200200- params.set('include_cards', '1');
201201- params.set('include_ext_alt_text', 'true');
202202- params.set('include_ext_limited_action_results', 'false');
203203- params.set('include_quote_count', 'true');
204204- params.set('include_reply_count', '1');
205205- params.set('tweet_mode', 'extended');
206206- params.set('include_ext_collab_control', 'true');
207207- params.set('include_ext_views', 'true');
208208- params.set('include_entities', 'true');
209209- params.set('include_user_entities', 'true');
210210- params.set('include_ext_media_color', 'true');
211211- params.set('include_ext_media_availability', 'true');
212212- params.set('include_ext_sensitive_media_warning', 'true');
213213- params.set('include_ext_trusted_friends_metadata', 'true');
214214- params.set('send_error_codes', 'true');
215215- params.set('simple_quoted_tweet', 'true');
216216- params.set('include_tweet_replies', `${includeTweetReplies}`);
217217- params.set(
218218- 'ext',
219219- 'mediaStats,highlightedLabel,hasNftAvatar,voiceInfo,birdwatchPivot,enrichments,superFollowMetadata,unmentionInfo,editControl,collab_control,vibe',
220220- );
221221- return params;
222222-}
···11-import { ApiError } from './errors';
22-import { ErrorRateLimitStrategy } from './rate-limit';
33-44-test('error rate limit strategy throws error when triggered', async () => {
55- const strategy = new ErrorRateLimitStrategy();
66-77- await expect(() =>
88- strategy.onRateLimit({
99- fetchParameters: ['/', {}],
1010- response: {
1111- headers: new Headers(),
1212- ok: false,
1313- redirected: false,
1414- status: 0,
1515- statusText: '',
1616- type: 'basic',
1717- url: '',
1818- clone: function (): Response {
1919- throw new Error('Function not implemented.');
2020- },
2121- body: null,
2222- bodyUsed: false,
2323- arrayBuffer: function (): Promise<ArrayBuffer> {
2424- throw new Error('Function not implemented.');
2525- },
2626- blob: function (): Promise<Blob> {
2727- throw new Error('Function not implemented.');
2828- },
2929- formData: function (): Promise<FormData> {
3030- throw new Error('Function not implemented.');
3131- },
3232- json: function (): Promise<any> {
3333- throw new Error('Function not implemented.');
3434- },
3535- text: function (): Promise<string> {
3636- throw new Error('Function not implemented.');
3737- },
3838- },
3939- }),
4040- ).rejects.toThrow(ApiError);
4141-});
-85
twitter-scraper/src/rate-limit.ts
···11-import { FetchParameters } from './api-types';
22-import { ApiError } from './errors';
33-import debug from 'debug';
44-55-const log = debug('twitter-scraper:rate-limit');
66-77-/**
88- * Information about a rate-limiting event. Both the request and response
99- * information are provided.
1010- */
1111-export interface RateLimitEvent {
1212- /** The complete arguments that were passed to the fetch function. */
1313- fetchParameters: FetchParameters;
1414- /** The failing HTTP response. */
1515- response: Response;
1616-}
1717-1818-/**
1919- * The public interface for all rate-limiting strategies. Library consumers are
2020- * welcome to provide their own implementations of this interface in the Scraper
2121- * constructor options.
2222- *
2323- * The {@link RateLimitEvent} object contains both the request and response
2424- * information associated with the event.
2525- *
2626- * @example
2727- * import { Scraper, RateLimitStrategy } from "@the-convocation/twitter-scraper";
2828- *
2929- * // A custom rate-limiting implementation that just logs request/response information.
3030- * class ConsoleLogRateLimitStrategy implements RateLimitStrategy {
3131- * async onRateLimit(event: RateLimitEvent): Promise<void> {
3232- * console.log(event.fetchParameters, event.response);
3333- * }
3434- * }
3535- *
3636- * const scraper = new Scraper({
3737- * rateLimitStrategy: new ConsoleLogRateLimitStrategy(),
3838- * });
3939- */
4040-export interface RateLimitStrategy {
4141- /**
4242- * Called when the scraper is rate limited.
4343- * @param event The event information, including the request and response info.
4444- */
4545- onRateLimit(event: RateLimitEvent): Promise<void>;
4646-}
4747-4848-/**
4949- * A rate-limiting strategy that simply waits for the current rate limit period to expire.
5050- * This has been known to take up to 13 minutes, in some cases.
5151- */
5252-export class WaitingRateLimitStrategy implements RateLimitStrategy {
5353- async onRateLimit({ response: res }: RateLimitEvent): Promise<void> {
5454- /*
5555- Known headers at this point:
5656- - x-rate-limit-limit: Maximum number of requests per time period?
5757- - x-rate-limit-reset: UNIX timestamp when the current rate limit will be reset.
5858- - x-rate-limit-remaining: Number of requests remaining in current time period?
5959- */
6060- const xRateLimitLimit = res.headers.get('x-rate-limit-limit');
6161- const xRateLimitRemaining = res.headers.get('x-rate-limit-remaining');
6262- const xRateLimitReset = res.headers.get('x-rate-limit-reset');
6363-6464- log(
6565- `Rate limit event: limit=${xRateLimitLimit}, remaining=${xRateLimitRemaining}, reset=${xRateLimitReset}`,
6666- );
6767-6868- if (xRateLimitRemaining == '0' && xRateLimitReset) {
6969- const currentTime = new Date().valueOf() / 1000;
7070- const timeDeltaMs = 1000 * (parseInt(xRateLimitReset) - currentTime);
7171-7272- // I have seen this block for 800s (~13 *minutes*)
7373- await new Promise((resolve) => setTimeout(resolve, timeDeltaMs));
7474- }
7575- }
7676-}
7777-7878-/**
7979- * A rate-limiting strategy that throws an {@link ApiError} when a rate limiting event occurs.
8080- */
8181-export class ErrorRateLimitStrategy implements RateLimitStrategy {
8282- async onRateLimit({ response: res }: RateLimitEvent): Promise<void> {
8383- throw await ApiError.fromResponse(res);
8484- }
8585-}
-59
twitter-scraper/src/relationships.test.ts
···11-import { getScraper } from './test-utils';
22-33-test('scraper can get profile followers', async () => {
44- const scraper = await getScraper();
55-66- const seenProfiles = new Map<string, boolean>();
77- const maxProfiles = 50;
88- let nProfiles = 0;
99-1010- const profiles = await scraper.getFollowers(
1111- '1425600122885394432',
1212- maxProfiles,
1313- );
1414-1515- for await (const profile of profiles) {
1616- nProfiles++;
1717-1818- const id = profile.userId;
1919- expect(id).toBeTruthy();
2020-2121- if (id != null) {
2222- expect(seenProfiles.has(id)).toBeFalsy();
2323- seenProfiles.set(id, true);
2424- }
2525-2626- expect(profile.username).toBeTruthy();
2727- }
2828-2929- expect(nProfiles).toEqual(maxProfiles);
3030-});
3131-3232-test('scraper can get profile following', async () => {
3333- const scraper = await getScraper();
3434-3535- const seenProfiles = new Map<string, boolean>();
3636- const maxProfiles = 50;
3737- let nProfiles = 0;
3838-3939- const profiles = await scraper.getFollowing(
4040- '1425600122885394432',
4141- maxProfiles,
4242- );
4343-4444- for await (const profile of profiles) {
4545- nProfiles++;
4646-4747- const id = profile.userId;
4848- expect(id).toBeTruthy();
4949-5050- if (id != null) {
5151- expect(seenProfiles.has(id)).toBeFalsy();
5252- seenProfiles.set(id, true);
5353- }
5454-5555- expect(profile.username).toBeTruthy();
5656- }
5757-5858- expect(nProfiles).toEqual(maxProfiles);
5959-});
···11-import { Cookie } from 'tough-cookie';
22-import { bearerToken, FetchTransformOptions, RequestApiResult } from './api';
33-import { TwitterAuth, TwitterAuthOptions, TwitterGuestAuth } from './auth';
44-import { FlowSubtaskHandler, TwitterUserAuth } from './auth-user';
55-import { getProfile, getUserIdByScreenName, Profile } from './profile';
66-import {
77- fetchSearchProfiles,
88- fetchSearchTweets,
99- SearchMode,
1010- searchProfiles,
1111- searchTweets,
1212-} from './search';
1313-import {
1414- fetchProfileFollowing,
1515- fetchProfileFollowers,
1616- getFollowing,
1717- getFollowers,
1818-} from './relationships';
1919-import { QueryProfilesResponse, QueryTweetsResponse } from './timeline-v1';
2020-import { getTrends } from './trends';
2121-import {
2222- Tweet,
2323- getTweetAnonymous,
2424- getTweets,
2525- getLatestTweet,
2626- getLikedTweets,
2727- getTweetWhere,
2828- getTweetsWhere,
2929- getTweetsByUserId,
3030- TweetQuery,
3131- getTweet,
3232- fetchListTweets,
3333- getTweetsAndRepliesByUserId,
3434- getTweetsAndReplies,
3535- fetchLikedTweets,
3636-} from './tweets';
3737-import fetch from 'cross-fetch';
3838-import { RateLimitStrategy } from './rate-limit';
3939-import {
4040- DmConversationTimeline,
4141- DmInbox,
4242- DmMessageEntry,
4343- DmCursorOptions,
4444- getDmConversation,
4545- getDmMessages,
4646- getDmInbox,
4747- findDmConversationsByUserId,
4848- DmConversation,
4949-} from './direct-messages';
5050-5151-const twUrl = 'https://x.com';
5252-5353-export interface ScraperOptions {
5454- /**
5555- * An alternative fetch function to use instead of the default fetch function. This may be useful
5656- * in nonstandard runtime environments, such as edge workers.
5757- */
5858- fetch: typeof fetch;
5959-6060- /**
6161- * Additional options that control how requests and responses are processed. This can be used to
6262- * proxy requests through other hosts, for example.
6363- */
6464- transform: Partial<FetchTransformOptions>;
6565-6666- /**
6767- * A handling strategy for rate limits (HTTP 429).
6868- */
6969- rateLimitStrategy: RateLimitStrategy;
7070-7171- /**
7272- * Experimental features that may be added, changed, or removed at any time. Use with caution.
7373- */
7474- experimental: {
7575- /**
7676- * Enables the generation of the `x-client-transaction-id` header on requests. This may resolve some errors.
7777- */
7878- xClientTransactionId: boolean;
7979- /**
8080- * Enables the generation of the `x-xp-forwarded-for` header on requests. This may resolve some errors.
8181- */
8282- xpff: boolean;
8383- };
8484-}
8585-8686-/**
8787- * An interface to Twitter's undocumented API.
8888- * - Reusing Scraper objects is recommended to minimize the time spent authenticating unnecessarily.
8989- */
9090-export class Scraper {
9191- private auth!: TwitterAuth;
9292- private authTrends!: TwitterAuth;
9393- private token: string;
9494-9595- /**
9696- * Creates a new Scraper object.
9797- * - Scrapers maintain their own guest tokens for Twitter's internal API.
9898- * - Reusing Scraper objects is recommended to minimize the time spent authenticating unnecessarily.
9999- */
100100- constructor(private readonly options?: Partial<ScraperOptions>) {
101101- this.token = bearerToken;
102102- this.useGuestAuth();
103103- }
104104-105105- /**
106106- * Registers a subtask handler for the given subtask ID. This
107107- * will override any existing handler for the same subtask.
108108- * @param subtaskId The ID of the subtask to register the handler for.
109109- * @param subtaskHandler The handler function to register.
110110- */
111111- public registerAuthSubtaskHandler(
112112- subtaskId: string,
113113- subtaskHandler: FlowSubtaskHandler,
114114- ): void {
115115- if (this.auth instanceof TwitterUserAuth) {
116116- this.auth.registerSubtaskHandler(subtaskId, subtaskHandler);
117117- }
118118-119119- if (this.authTrends instanceof TwitterUserAuth) {
120120- this.authTrends.registerSubtaskHandler(subtaskId, subtaskHandler);
121121- }
122122- }
123123-124124- /**
125125- * Initializes auth properties using a guest token.
126126- * Used when creating a new instance of this class, and when logging out.
127127- * @internal
128128- */
129129- private useGuestAuth() {
130130- this.auth = new TwitterGuestAuth(this.token, this.getAuthOptions());
131131- this.authTrends = new TwitterGuestAuth(this.token, this.getAuthOptions());
132132- }
133133-134134- /**
135135- * Fetches a Twitter profile.
136136- * @param username The Twitter username of the profile to fetch, without an `@` at the beginning.
137137- * @returns The requested {@link Profile}.
138138- */
139139- public async getProfile(username: string): Promise<Profile> {
140140- const res = await getProfile(username, this.auth);
141141- return this.handleResponse(res);
142142- }
143143-144144- /**
145145- * Fetches the user ID corresponding to the provided screen name.
146146- * @param screenName The Twitter screen name of the profile to fetch.
147147- * @returns The ID of the corresponding account.
148148- */
149149- public async getUserIdByScreenName(screenName: string): Promise<string> {
150150- const res = await getUserIdByScreenName(screenName, this.auth);
151151- return this.handleResponse(res);
152152- }
153153-154154- /**
155155- * Fetches tweets from Twitter.
156156- * @param query The search query. Any Twitter-compatible query format can be used.
157157- * @param maxTweets The maximum number of tweets to return.
158158- * @param includeReplies Whether or not replies should be included in the response.
159159- * @param searchMode The category filter to apply to the search. Defaults to `Top`.
160160- * @returns An {@link AsyncGenerator} of tweets matching the provided filters.
161161- */
162162- public searchTweets(
163163- query: string,
164164- maxTweets: number,
165165- searchMode: SearchMode = SearchMode.Top,
166166- ): AsyncGenerator<Tweet, void> {
167167- return searchTweets(query, maxTweets, searchMode, this.auth);
168168- }
169169-170170- /**
171171- * Fetches profiles from Twitter.
172172- * @param query The search query. Any Twitter-compatible query format can be used.
173173- * @param maxProfiles The maximum number of profiles to return.
174174- * @returns An {@link AsyncGenerator} of tweets matching the provided filter(s).
175175- */
176176- public searchProfiles(
177177- query: string,
178178- maxProfiles: number,
179179- ): AsyncGenerator<Profile, void> {
180180- return searchProfiles(query, maxProfiles, this.auth);
181181- }
182182-183183- /**
184184- * Fetches tweets from Twitter.
185185- * @param query The search query. Any Twitter-compatible query format can be used.
186186- * @param maxTweets The maximum number of tweets to return.
187187- * @param includeReplies Whether or not replies should be included in the response.
188188- * @param searchMode The category filter to apply to the search. Defaults to `Top`.
189189- * @param cursor The search cursor, which can be passed into further requests for more results.
190190- * @returns A page of results, containing a cursor that can be used in further requests.
191191- */
192192- public fetchSearchTweets(
193193- query: string,
194194- maxTweets: number,
195195- searchMode: SearchMode,
196196- cursor?: string,
197197- ): Promise<QueryTweetsResponse> {
198198- return fetchSearchTweets(query, maxTweets, searchMode, this.auth, cursor);
199199- }
200200-201201- /**
202202- * Fetches profiles from Twitter.
203203- * @param query The search query. Any Twitter-compatible query format can be used.
204204- * @param maxProfiles The maximum number of profiles to return.
205205- * @param cursor The search cursor, which can be passed into further requests for more results.
206206- * @returns A page of results, containing a cursor that can be used in further requests.
207207- */
208208- public fetchSearchProfiles(
209209- query: string,
210210- maxProfiles: number,
211211- cursor?: string,
212212- ): Promise<QueryProfilesResponse> {
213213- return fetchSearchProfiles(query, maxProfiles, this.auth, cursor);
214214- }
215215-216216- /**
217217- * Fetches list tweets from Twitter.
218218- * @param listId The list id
219219- * @param maxTweets The maximum number of tweets to return.
220220- * @param cursor The search cursor, which can be passed into further requests for more results.
221221- * @returns A page of results, containing a cursor that can be used in further requests.
222222- */
223223- public fetchListTweets(
224224- listId: string,
225225- maxTweets: number,
226226- cursor?: string,
227227- ): Promise<QueryTweetsResponse> {
228228- return fetchListTweets(listId, maxTweets, cursor, this.auth);
229229- }
230230-231231- /**
232232- * Fetch the tweets a user has liked
233233- * @param userId The user whose liked tweets should be returned
234234- * @param maxTweets The maximum number of tweets to return.
235235- * @param cursor The search cursor, which can be passed into further requests for more results.
236236- * @returns A page of results, containing a cursor that can be used in further requests.
237237- */
238238- public fetchLikedTweets(
239239- userId: string,
240240- maxTweets: number,
241241- cursor?: string,
242242- ): Promise<QueryTweetsResponse> {
243243- return fetchLikedTweets(userId, maxTweets, cursor, this.auth);
244244- }
245245-246246- /**
247247- * Fetch the profiles a user is following
248248- * @param userId The user whose following should be returned
249249- * @param maxProfiles The maximum number of profiles to return.
250250- * @returns An {@link AsyncGenerator} of following profiles for the provided user.
251251- */
252252- public getFollowing(
253253- userId: string,
254254- maxProfiles: number,
255255- ): AsyncGenerator<Profile, void> {
256256- return getFollowing(userId, maxProfiles, this.auth);
257257- }
258258-259259- /**
260260- * Fetch the profiles that follow a user
261261- * @param userId The user whose followers should be returned
262262- * @param maxProfiles The maximum number of profiles to return.
263263- * @returns An {@link AsyncGenerator} of profiles following the provided user.
264264- */
265265- public getFollowers(
266266- userId: string,
267267- maxProfiles: number,
268268- ): AsyncGenerator<Profile, void> {
269269- return getFollowers(userId, maxProfiles, this.auth);
270270- }
271271-272272- /**
273273- * Fetches following profiles from Twitter.
274274- * @param userId The user whose following should be returned
275275- * @param maxProfiles The maximum number of profiles to return.
276276- * @param cursor The search cursor, which can be passed into further requests for more results.
277277- * @returns A page of results, containing a cursor that can be used in further requests.
278278- */
279279- public fetchProfileFollowing(
280280- userId: string,
281281- maxProfiles: number,
282282- cursor?: string,
283283- ): Promise<QueryProfilesResponse> {
284284- return fetchProfileFollowing(userId, maxProfiles, this.auth, cursor);
285285- }
286286-287287- /**
288288- * Fetches profile followers from Twitter.
289289- * @param userId The user whose following should be returned
290290- * @param maxProfiles The maximum number of profiles to return.
291291- * @param cursor The search cursor, which can be passed into further requests for more results.
292292- * @returns A page of results, containing a cursor that can be used in further requests.
293293- */
294294- public fetchProfileFollowers(
295295- userId: string,
296296- maxProfiles: number,
297297- cursor?: string,
298298- ): Promise<QueryProfilesResponse> {
299299- return fetchProfileFollowers(userId, maxProfiles, this.auth, cursor);
300300- }
301301-302302- /**
303303- * Fetches the current trends from Twitter.
304304- * @returns The current list of trends.
305305- */
306306- public getTrends(): Promise<string[]> {
307307- return getTrends(this.authTrends);
308308- }
309309-310310- /**
311311- * Fetches tweets from a Twitter user.
312312- * @param user The user whose tweets should be returned.
313313- * @param maxTweets The maximum number of tweets to return. Defaults to `200`.
314314- * @returns An {@link AsyncGenerator} of tweets from the provided user.
315315- */
316316- public getTweets(user: string, maxTweets = 200): AsyncGenerator<Tweet> {
317317- return getTweets(user, maxTweets, this.auth);
318318- }
319319-320320- /**
321321- * Fetches liked tweets from a Twitter user. Requires authentication.
322322- * @param user The user whose likes should be returned.
323323- * @param maxTweets The maximum number of tweets to return. Defaults to `200`.
324324- * @returns An {@link AsyncGenerator} of liked tweets from the provided user.
325325- */
326326- public getLikedTweets(user: string, maxTweets = 200): AsyncGenerator<Tweet> {
327327- return getLikedTweets(user, maxTweets, this.auth);
328328- }
329329-330330- /**
331331- * Fetches tweets from a Twitter user using their ID.
332332- * @param userId The user whose tweets should be returned.
333333- * @param maxTweets The maximum number of tweets to return. Defaults to `200`.
334334- * @returns An {@link AsyncGenerator} of tweets from the provided user.
335335- */
336336- public getTweetsByUserId(
337337- userId: string,
338338- maxTweets = 200,
339339- ): AsyncGenerator<Tweet, void> {
340340- return getTweetsByUserId(userId, maxTweets, this.auth);
341341- }
342342-343343- /**
344344- * Fetches tweets and replies from a Twitter user.
345345- * @param user The user whose tweets should be returned.
346346- * @param maxTweets The maximum number of tweets to return. Defaults to `200`.
347347- * @returns An {@link AsyncGenerator} of tweets from the provided user.
348348- */
349349- public getTweetsAndReplies(
350350- user: string,
351351- maxTweets = 200,
352352- ): AsyncGenerator<Tweet> {
353353- return getTweetsAndReplies(user, maxTweets, this.auth);
354354- }
355355-356356- /**
357357- * Fetches tweets and replies from a Twitter user using their ID.
358358- * @param userId The user whose tweets should be returned.
359359- * @param maxTweets The maximum number of tweets to return. Defaults to `200`.
360360- * @returns An {@link AsyncGenerator} of tweets from the provided user.
361361- */
362362- public getTweetsAndRepliesByUserId(
363363- userId: string,
364364- maxTweets = 200,
365365- ): AsyncGenerator<Tweet, void> {
366366- return getTweetsAndRepliesByUserId(userId, maxTweets, this.auth);
367367- }
368368-369369- /**
370370- * Fetches the first tweet matching the given query.
371371- *
372372- * Example:
373373- * ```js
374374- * const timeline = scraper.getTweets('user', 200);
375375- * const retweet = await scraper.getTweetWhere(timeline, { isRetweet: true });
376376- * ```
377377- * @param tweets The {@link AsyncIterable} of tweets to search through.
378378- * @param query A query to test **all** tweets against. This may be either an
379379- * object of key/value pairs or a predicate. If this query is an object, all
380380- * key/value pairs must match a {@link Tweet} for it to be returned. If this query
381381- * is a predicate, it must resolve to `true` for a {@link Tweet} to be returned.
382382- * - All keys are optional.
383383- * - If specified, the key must be implemented by that of {@link Tweet}.
384384- */
385385- public getTweetWhere(
386386- tweets: AsyncIterable<Tweet>,
387387- query: TweetQuery,
388388- ): Promise<Tweet | null> {
389389- return getTweetWhere(tweets, query);
390390- }
391391-392392- /**
393393- * Fetches all tweets matching the given query.
394394- *
395395- * Example:
396396- * ```js
397397- * const timeline = scraper.getTweets('user', 200);
398398- * const retweets = await scraper.getTweetsWhere(timeline, { isRetweet: true });
399399- * ```
400400- * @param tweets The {@link AsyncIterable} of tweets to search through.
401401- * @param query A query to test **all** tweets against. This may be either an
402402- * object of key/value pairs or a predicate. If this query is an object, all
403403- * key/value pairs must match a {@link Tweet} for it to be returned. If this query
404404- * is a predicate, it must resolve to `true` for a {@link Tweet} to be returned.
405405- * - All keys are optional.
406406- * - If specified, the key must be implemented by that of {@link Tweet}.
407407- */
408408- public getTweetsWhere(
409409- tweets: AsyncIterable<Tweet>,
410410- query: TweetQuery,
411411- ): Promise<Tweet[]> {
412412- return getTweetsWhere(tweets, query);
413413- }
414414-415415- /**
416416- * Fetches the most recent tweet from a Twitter user.
417417- * @param user The user whose latest tweet should be returned.
418418- * @param includeRetweets Whether or not to include retweets. Defaults to `false`.
419419- * @returns The {@link Tweet} object or `null`/`undefined` if it couldn't be fetched.
420420- */
421421- public getLatestTweet(
422422- user: string,
423423- includeRetweets = false,
424424- max = 200,
425425- ): Promise<Tweet | null | void> {
426426- return getLatestTweet(user, includeRetweets, max, this.auth);
427427- }
428428-429429- /**
430430- * Fetches a single tweet.
431431- * @param id The ID of the tweet to fetch.
432432- * @returns The {@link Tweet} object, or `null` if it couldn't be fetched.
433433- */
434434- public getTweet(id: string): Promise<Tweet | null> {
435435- if (this.auth instanceof TwitterUserAuth) {
436436- return getTweet(id, this.auth);
437437- } else {
438438- return getTweetAnonymous(id, this.auth);
439439- }
440440- }
441441-442442- /**
443443- * Retrieves the direct message inbox for the authenticated user.
444444- *
445445- * @return A promise that resolves to an object representing the direct message inbox.
446446- */
447447- public getDmInbox(): Promise<DmInbox> {
448448- return getDmInbox(this.auth);
449449- }
450450-451451- /**
452452- * Retrieves the direct message conversation for the specified conversation ID.
453453- *
454454- * @param conversationId - The unique identifier of the DM conversation to retrieve.
455455- * @param cursor - Use `maxId` to get messages before a message ID (older messages), or `minId` to get messages after a message ID (newer messages).
456456- * @return A promise that resolves to the timeline of the DM conversation.
457457- */
458458- public getDmConversation(
459459- conversationId: string,
460460- cursor?: DmCursorOptions,
461461- ): Promise<DmConversationTimeline> {
462462- return getDmConversation(conversationId, cursor, this.auth);
463463- }
464464-465465- /**
466466- * Retrieves direct messages from a specific conversation.
467467- *
468468- * @param conversationId - The unique identifier of the conversation to fetch messages from.
469469- * @param [maxMessages=20] - The maximum number of messages to retrieve per request.
470470- * @param cursor - Use `maxId` to get messages before a message ID (older messages), or `minId` to get messages after a message ID (newer messages).
471471- * @returns An {@link AsyncGenerator} of messages from the provided conversation.
472472- */
473473- public getDmMessages(
474474- conversationId: string,
475475- maxMessages = 20,
476476- cursor?: DmCursorOptions,
477477- ): AsyncGenerator<DmMessageEntry, void> {
478478- return getDmMessages(conversationId, maxMessages, cursor, this.auth);
479479- }
480480-481481- /**
482482- * Retrieves a list of direct message conversations for a specific user based on their user ID.
483483- *
484484- * @param inbox - The DM inbox containing all available conversations.
485485- * @param userId - The unique identifier of the user whose DM conversations are to be retrieved.
486486- * @return An array of DM conversations associated with the specified user ID.
487487- */
488488- public findDmConversationsByUserId(
489489- inbox: DmInbox,
490490- userId: string,
491491- ): DmConversation[] {
492492- return findDmConversationsByUserId(inbox, userId);
493493- }
494494-495495- /**
496496- * Returns if the scraper has a guest token. The token may not be valid.
497497- * @returns `true` if the scraper has a guest token; otherwise `false`.
498498- */
499499- public hasGuestToken(): boolean {
500500- return this.auth.hasToken() || this.authTrends.hasToken();
501501- }
502502-503503- /**
504504- * Returns if the scraper is logged in as a real user.
505505- * @returns `true` if the scraper is logged in with a real user account; otherwise `false`.
506506- */
507507- public async isLoggedIn(): Promise<boolean> {
508508- return (
509509- (await this.auth.isLoggedIn()) && (await this.authTrends.isLoggedIn())
510510- );
511511- }
512512-513513- /**
514514- * Login to Twitter as a real Twitter account. This enables running
515515- * searches.
516516- * @param username The username of the Twitter account to login with.
517517- * @param password The password of the Twitter account to login with.
518518- * @param email The email to log in with, if you have email confirmation enabled.
519519- * @param twoFactorSecret The secret to generate two factor authentication tokens with, if you have two factor authentication enabled.
520520- */
521521- public async login(
522522- username: string,
523523- password: string,
524524- email?: string,
525525- twoFactorSecret?: string,
526526- ): Promise<void> {
527527- // Swap in a real authorizer for all requests
528528- const userAuth = new TwitterUserAuth(this.token, this.getAuthOptions());
529529- await userAuth.login(username, password, email, twoFactorSecret);
530530- this.auth = userAuth;
531531- this.authTrends = userAuth;
532532- }
533533-534534- /**
535535- * Log out of Twitter.
536536- */
537537- public async logout(): Promise<void> {
538538- await this.auth.logout();
539539- await this.authTrends.logout();
540540-541541- // Swap in guest authorizers for all requests
542542- this.useGuestAuth();
543543- }
544544-545545- /**
546546- * Retrieves all cookies for the current session.
547547- * @returns All cookies for the current session.
548548- */
549549- public async getCookies(): Promise<Cookie[]> {
550550- return await this.authTrends
551551- .cookieJar()
552552- .getCookies(
553553- typeof document !== 'undefined' ? document.location.toString() : twUrl,
554554- );
555555- }
556556-557557- /**
558558- * Set cookies for the current session.
559559- * @param cookies The cookies to set for the current session.
560560- */
561561- public async setCookies(cookies: (string | Cookie)[]): Promise<void> {
562562- const userAuth = new TwitterUserAuth(this.token, this.getAuthOptions());
563563- for (const cookie of cookies) {
564564- await userAuth.cookieJar().setCookie(cookie, twUrl);
565565- }
566566-567567- this.auth = userAuth;
568568- this.authTrends = userAuth;
569569- }
570570-571571- /**
572572- * Clear all cookies for the current session.
573573- */
574574- public async clearCookies(): Promise<void> {
575575- await this.auth.cookieJar().removeAllCookies();
576576- await this.authTrends.cookieJar().removeAllCookies();
577577- }
578578-579579- /**
580580- * Sets the optional cookie to be used in requests.
581581- * @param _cookie The cookie to be used in requests.
582582- * @deprecated This function no longer represents any part of Twitter's auth flow.
583583- * @returns This scraper instance.
584584- */
585585- // eslint-disable-next-line @typescript-eslint/no-unused-vars
586586- public withCookie(_cookie: string): Scraper {
587587- console.warn(
588588- 'Warning: Scraper#withCookie is deprecated and will be removed in a later version. Use Scraper#login or Scraper#setCookies instead.',
589589- );
590590- return this;
591591- }
592592-593593- /**
594594- * Sets the optional CSRF token to be used in requests.
595595- * @param _token The CSRF token to be used in requests.
596596- * @deprecated This function no longer represents any part of Twitter's auth flow.
597597- * @returns This scraper instance.
598598- */
599599- // eslint-disable-next-line @typescript-eslint/no-unused-vars
600600- public withXCsrfToken(_token: string): Scraper {
601601- console.warn(
602602- 'Warning: Scraper#withXCsrfToken is deprecated and will be removed in a later version.',
603603- );
604604- return this;
605605- }
606606-607607- private getAuthOptions(): Partial<TwitterAuthOptions> {
608608- return {
609609- fetch: this.options?.fetch,
610610- transform: this.options?.transform,
611611- rateLimitStrategy: this.options?.rateLimitStrategy,
612612- experimental: {
613613- xClientTransactionId: this.options?.experimental?.xClientTransactionId,
614614- xpff: this.options?.experimental?.xpff,
615615- },
616616- };
617617- }
618618-619619- private handleResponse<T>(res: RequestApiResult<T>): T {
620620- if (!res.success) {
621621- throw res.err;
622622- }
623623-624624- return res.value;
625625- }
626626-}
···11-export type NonNullableField<T, K extends keyof T> = {
22- [P in K]-?: T[P];
33-} & T;
44-55-export function isFieldDefined<T, K extends keyof T>(key: K) {
66- return function (value: T): value is NonNullableField<T, K> {
77- return isDefined(value[key]);
88- };
99-}
1010-1111-export function isDefined<T>(value: T | null | undefined): value is T {
1212- return value != null;
1313-}
-176
twitter-scraper/src/xctxid.ts
···11-import fetch from 'cross-fetch';
22-import debug from 'debug';
33-44-const log = debug('twitter-scraper:xctxid');
55-66-// @ts-expect-error import type annotation ("the current file is a CommonJS module")
77-type LinkeDOM = typeof import('linkedom');
88-99-let linkedom: LinkeDOM | null = null;
1010-async function linkedomImport(): Promise<LinkeDOM> {
1111- if (!linkedom) {
1212- const mod = await import('linkedom');
1313- linkedom = mod;
1414- return mod;
1515- }
1616- return linkedom;
1717-}
1818-1919-async function parseHTML(html: string): Promise<Window & typeof globalThis> {
2020- if (typeof window !== 'undefined') {
2121- const { defaultView } = new DOMParser().parseFromString(html, 'text/html');
2222- if (!defaultView) {
2323- throw new Error('Failed to get defaultView from parsed HTML.');
2424- }
2525- return defaultView;
2626- } else {
2727- const { DOMParser } = await linkedomImport();
2828- return new DOMParser().parseFromString(html, 'text/html').defaultView;
2929- }
3030-}
3131-3232-// Copied from https://github.com/Lqm1/x-client-transaction-id/blob/main/utils.ts with minor tweaks to support us passing a custom fetch function
3333-async function handleXMigration(fetchFn: typeof fetch): Promise<Document> {
3434- // Set headers to mimic a browser request
3535- const headers = {
3636- accept:
3737- 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
3838- 'accept-language': 'ja',
3939- 'cache-control': 'no-cache',
4040- pragma: 'no-cache',
4141- priority: 'u=0, i',
4242- 'sec-ch-ua':
4343- '"Google Chrome";v="135", "Not-A.Brand";v="8", "Chromium";v="135"',
4444- 'sec-ch-ua-mobile': '?0',
4545- 'sec-ch-ua-platform': '"Windows"',
4646- 'sec-fetch-dest': 'document',
4747- 'sec-fetch-mode': 'navigate',
4848- 'sec-fetch-site': 'none',
4949- 'sec-fetch-user': '?1',
5050- 'upgrade-insecure-requests': '1',
5151- 'user-agent':
5252- 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36',
5353- };
5454-5555- // Fetch X.com homepage
5656- const response = await fetchFn('https://x.com', {
5757- headers,
5858- });
5959-6060- if (!response.ok) {
6161- throw new Error(`Failed to fetch X homepage: ${response.statusText}`);
6262- }
6363-6464- const htmlText = await response.text();
6565-6666- // Parse HTML using linkedom
6767- let dom = await parseHTML(htmlText);
6868- let document = dom.window.document;
6969-7070- // Check for migration redirection links
7171- const migrationRedirectionRegex = new RegExp(
7272- '(http(?:s)?://(?:www\\.)?(twitter|x){1}\\.com(/x)?/migrate([/?])?tok=[a-zA-Z0-9%\\-_]+)+',
7373- 'i',
7474- );
7575-7676- const metaRefresh = document.querySelector("meta[http-equiv='refresh']");
7777- const metaContent = metaRefresh
7878- ? metaRefresh.getAttribute('content') || ''
7979- : '';
8080-8181- const migrationRedirectionUrl =
8282- migrationRedirectionRegex.exec(metaContent) ||
8383- migrationRedirectionRegex.exec(htmlText);
8484-8585- if (migrationRedirectionUrl) {
8686- // Follow redirection URL
8787- const redirectResponse = await fetch(migrationRedirectionUrl[0]);
8888-8989- if (!redirectResponse.ok) {
9090- throw new Error(
9191- `Failed to follow migration redirection: ${redirectResponse.statusText}`,
9292- );
9393- }
9494-9595- const redirectHtml = await redirectResponse.text();
9696- dom = await parseHTML(redirectHtml);
9797- document = dom.window.document;
9898- }
9999-100100- // Handle migration form if present
101101- const migrationForm =
102102- document.querySelector("form[name='f']") ||
103103- document.querySelector("form[action='https://x.com/x/migrate']");
104104-105105- if (migrationForm) {
106106- const url =
107107- migrationForm.getAttribute('action') || 'https://x.com/x/migrate';
108108- const method = migrationForm.getAttribute('method') || 'POST';
109109-110110- // Collect form input fields
111111- const requestPayload = new FormData();
112112-113113- const inputFields = migrationForm.querySelectorAll('input');
114114- for (const element of Array.from(inputFields)) {
115115- const name = element.getAttribute('name');
116116- const value = element.getAttribute('value');
117117- if (name && value) {
118118- requestPayload.append(name, value);
119119- }
120120- }
121121-122122- // Submit form using POST request
123123- const formResponse = await fetch(url, {
124124- method: method,
125125- body: requestPayload,
126126- headers,
127127- });
128128-129129- if (!formResponse.ok) {
130130- throw new Error(
131131- `Failed to submit migration form: ${formResponse.statusText}`,
132132- );
133133- }
134134-135135- const formHtml = await formResponse.text();
136136- dom = await parseHTML(formHtml);
137137- document = dom.window.document;
138138- }
139139-140140- // Return final DOM document
141141- return document;
142142-}
143143-144144-let ClientTransaction:
145145- | typeof import('x-client-transaction-id')['ClientTransaction']
146146- | null = null;
147147-async function clientTransaction(): Promise<
148148- typeof import('x-client-transaction-id')['ClientTransaction']
149149-> {
150150- if (!ClientTransaction) {
151151- const mod = await import('x-client-transaction-id');
152152- // eslint-disable-next-line @typescript-eslint/no-explicit-any
153153- ClientTransaction = mod.ClientTransaction as any;
154154- // eslint-disable-next-line @typescript-eslint/no-explicit-any
155155- return mod.ClientTransaction as any;
156156- }
157157- return ClientTransaction;
158158-}
159159-160160-export async function generateTransactionId(
161161- url: string,
162162- fetchFn: typeof fetch,
163163- method: 'GET' | 'POST',
164164-) {
165165- const parsedUrl = new URL(url);
166166- const path = parsedUrl.pathname;
167167-168168- log(`Generating transaction ID for ${method} ${path}`);
169169- const document = await handleXMigration(fetchFn);
170170- const ClientTransactionClass = await clientTransaction();
171171- const transaction = await ClientTransactionClass.create(document);
172172- const transactionId = await transaction.generateTransactionId(method, path);
173173- log(`Transaction ID: ${transactionId}`);
174174-175175- return transactionId;
176176-}
-103
twitter-scraper/src/xpff.ts
···11-import debug from 'debug';
22-33-const log = debug('twitter-scraper:xpff');
44-55-let isoCrypto: Crypto | null = null;
66-77-async function getCrypto(): Promise<Crypto> {
88- if (isoCrypto != null) {
99- return isoCrypto;
1010- }
1111-1212- // In Node.js, the global `crypto` object is only available from v19.0.0 onwards.
1313- // For earlier versions, we need to import the 'crypto' module.
1414- if (typeof crypto === 'undefined') {
1515- log('Global crypto is undefined, importing from crypto module...');
1616- const { webcrypto } = await import('crypto');
1717- // eslint-disable-next-line @typescript-eslint/no-explicit-any
1818- isoCrypto = webcrypto as any;
1919- // eslint-disable-next-line @typescript-eslint/no-explicit-any
2020- return webcrypto as any;
2121- }
2222- isoCrypto = crypto;
2323- return crypto;
2424-}
2525-2626-async function sha256(message: string): Promise<Uint8Array> {
2727- const msgBuffer = new TextEncoder().encode(message);
2828- const crypto = await getCrypto();
2929- const hashBuffer = await crypto.subtle.digest('SHA-256', msgBuffer);
3030- return new Uint8Array(hashBuffer);
3131-}
3232-3333-// https://stackoverflow.com/a/40031979
3434-function buf2hex(buffer: ArrayBuffer): string {
3535- return [...new Uint8Array(buffer)]
3636- .map((x) => x.toString(16).padStart(2, '0'))
3737- .join('');
3838-}
3939-4040-// Adapted from https://github.com/dsekz/twitter-x-xp-forwarded-for-header
4141-export class XPFFHeaderGenerator {
4242- constructor(private readonly seed: string) {}
4343-4444- private async deriveKey(guestId: string): Promise<Uint8Array> {
4545- const combined = `${this.seed}${guestId}`;
4646- const result = await sha256(combined);
4747- return result;
4848- }
4949-5050- async generateHeader(plaintext: string, guestId: string): Promise<string> {
5151- log(`Generating XPFF key for guest ID: ${guestId}`);
5252- const key = await this.deriveKey(guestId);
5353- const crypto = await getCrypto();
5454- const nonce = crypto.getRandomValues(new Uint8Array(12));
5555- const cipher = await crypto.subtle.importKey(
5656- 'raw',
5757- key as BufferSource,
5858- { name: 'AES-GCM' },
5959- false,
6060- ['encrypt'],
6161- );
6262- const encrypted = await crypto.subtle.encrypt(
6363- {
6464- name: 'AES-GCM',
6565- iv: nonce,
6666- },
6767- cipher,
6868- new TextEncoder().encode(plaintext),
6969- );
7070-7171- // Combine nonce and encrypted data
7272- const combined = new Uint8Array(nonce.length + encrypted.byteLength);
7373- combined.set(nonce);
7474- combined.set(new Uint8Array(encrypted), nonce.length);
7575- const result = buf2hex(combined.buffer);
7676-7777- log(`XPFF header generated for guest ID ${guestId}: ${result}`);
7878-7979- return result;
8080- }
8181-}
8282-8383-const xpffBaseKey =
8484- '0e6be1f1e21ffc33590b888fd4dc81b19713e570e805d4e5df80a493c9571a05';
8585-8686-function xpffPlain(): string {
8787- const timestamp = Date.now();
8888- return JSON.stringify({
8989- navigator_properties: {
9090- hasBeenActive: 'true',
9191- userAgent:
9292- 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36',
9393- webdriver: 'false',
9494- },
9595- created_at: timestamp,
9696- });
9797-}
9898-9999-export async function generateXPFFHeader(guestId: string): Promise<string> {
100100- const generator = new XPFFHeaderGenerator(xpffBaseKey);
101101- const plaintext = xpffPlain();
102102- return generator.generateHeader(plaintext, guestId);
103103-}