Rare chance of 404 when making more than one request per second

[wyj8bc5odzfl]

This is a follow up from the rejected pull request #15

To give a bit more context on how I encountered the issue (and why it is reasonable to expect it to occur again for someone else), here is what I was doing:
I have a script which calls the UsersByRestIds endpoint for a large amount of users.
I had one call that started at second 12.24, and then the next call started at second 12.85 (0.6 second gap).
Since both calls took place at the same second, the x-client-transaction-id contained the same timestamp.

The 12.24 call worked just fine, the 12.85 call got a 404.

Further examination led me to notice that both calls used the exact same x-client-transaction-id header.
When you inspect the code of generate_transaction_id(), you will notice that the only thing that could cause them to differ in this circumstance is the value of random_num.

Indeed, the second request was unlucky enough the use the same random_num value as the first.

Testing this further, I tried setting random_num to a fixed value, so that I could verify that Twitter rejects reuse of the header. I saw the pattern of first request works, second or third fails.
(As a note: I think that I was always experiencing a 404 on the second request originally, but when testing right now, I'm getting it on the third. I wonder if this is to make it a less likely problem for real Twitter usage, since they don't account for this in the client.)

In my pull request, I stated that:

Twitter returns a 404 if the value used for the XOR in generate_transaction_id() is used more than once in a given second.

and suggested ensuring that unique pairs of (time_now, random_num) are used when generating the x-client-transaction-id header. I still think that something along those lines would work, but there are some other aspects that could be taken into consideration:

• I have not tested if you can reuse a (time_now, random_num) value across different endpoints within a second, which, if you can, implies that it is only the full x-client-transaction-id that cannot be reused
• While you may not be able to reuse a header immediately, you may be able to use it again after some short time has passed. I think I saw this occurring when I was first trying to understand the header, but I haven't been back to study it more.

[iSarabjitDhiman]

Hey @wyj8bc5odzfl time.time() generates a timestamp which is milliseconds based. And then there is this random number and some other factors which are responsible for generating the tid. If for some reason the random number is same, the timestamp will not be. There are rare chances when the tid is going to be the as same as the previous one, but that shouldn't be a problem.
On top of that each tid can be reused 5-7 times or even more, not sure how many times but I have seen people reusing it. The reason you encountered 404 might not even have anything to do with this, there might be some other reason. Its totally normal if you get a 404 every once in hundred/thousand times, even twitter itself sometimes show 404 error page if u try to refresh too frequently.

Let's just wait and see if there is anyone else facing the same issue.

[wyj8bc5odzfl]

I fully agree it's going to be quite a rare problem in normal use (especially now that it fails out on the third reuse), but it is entirely repeatable for me if I force the unlucky conditions. Your call, of course, if you feel it warrants any special handling or not. At least it won't be a mystery to us! 😄

time.time() generates a timestamp which is milliseconds based.

That's true for time.time(), but you are wrapping it in math.floor() inside of generate_transaction_id(), which is keeping the seconds only:

>>> timestamp = time.time()
>>> timestamp
1746616026.667256
>>> math.floor(timestamp)
1746616026

Note that the docs for time.time() say Return[s] the time in seconds since the epoch as a floating-point number.

You can run math.floor((time.time() * 1000 - 1682924400 * 1000) / 1000) in the Python CLI multiple times if you'd like to see what is actually going into the tid. Right now, it's 63692399, and now that I've typed that, it's 63692407 (8 seconds elapsed).

(math.floor((time.time() * 1000 - 1682924400 * 1000) / 1000) is the same as math.floor(time.time() - 1682924400), if it wasn't obvious.)

On top of that each tid can be reused 5-7 times or even more, not sure how many times but I have seen people reusing it.

I expect that this might be related to my last point:

While you may not be able to reuse a header immediately, you may be able to use it again after some short time has passed.

Have people been able to reuse the tid successfully without much time between requests, on an endpoint that requires authentication? I suspect that it's likely that those are the conditions required for this issue to be noticeable.

[iSarabjitDhiman]

@wyj8bc5odzfl lets connect on discord https://discord.gg/pHY6CU5Ke4
Some of the users there are reusing tids, maybe you can discuss with them and then we decide something.

Lets connect.

[wyj8bc5odzfl]

I'm sorry, but I prefer not to use Discord. I'm happy to provide a basic script that should demonstrate the problem, though.

First, in order to simulate the issue without chance getting in the way, we need to be able to manipulate random_num in generate_transaction_id(), so that function needs to be modified slightly:
Add optional keyword parameter random_num to the function definition:

-    def generate_transaction_id(self, method: str, path: str, home_page_response: Optional[bs4.BeautifulSoup] = None, key: Optional[str] = None, animation_key: Optional[str] = None, time_now: Optional[int] = None) -> str:
+    def generate_transaction_id(self, method: str, path: str, home_page_response: Optional[bs4.BeautifulSoup] = None, key: Optional[str] = None, animation_key: Optional[str] = None, time_now: Optional[int] = None, random_num: Optional[int] = None) -> str:

Allow it to override the random value of random_num:

-        random_num = random.randint(0, 255)
+        random_num = random_num or random.randint(0, 255)

Test script:

import math
import random
import time
import bs4
import requests
from x_client_transaction.utils import generate_headers, handle_x_migration, get_ondemand_file_url

from urllib.parse import urlparse
from x_client_transaction import ClientTransaction

session = requests.Session()
session.headers = generate_headers()

# !!!! Set the cookie as appropriate !!!!
session.headers["Cookie"] = 'ct0=VALUE_OF_ct0_COOKIE; auth_token=VALUE_OF_auth_token_COOKIE;'

home_page = session.get(url="https://x.com")
home_page_response = bs4.BeautifulSoup(home_page.content, 'html.parser')

ondemand_file_url = get_ondemand_file_url(response=home_page_response)
ondemand_file = session.get(url=ondemand_file_url)
ondemand_file_response = bs4.BeautifulSoup(ondemand_file.content, 'html.parser')

ct = ClientTransaction(home_page_response=home_page_response, ondemand_file_response=ondemand_file_response)


api_url = """https://x.com/i/api/graphql/XArUHrueMW0KQdZUdqidrA/UsersByRestIds?variables=%7B%22userIds%22%3A+%5B%2213334762%22%5D%7D&features=%7B%22profile_label_improvements_pcf_label_in_post_enabled%22%3A+true%2C+%22rweb_tipjar_consumption_enabled%22%3A+true%2C+%22verified_phone_label_enabled%22%3A+false%2C+%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3A+false%2C+%22responsive_web_graphql_timeline_navigation_enabled%22%3A+true%7D"""
api_url_path = urlparse(api_url).path


session.headers["authorization"] = "Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs=1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA"

# !!!! Set this header as appropriate !!!!
session.headers["x-csrf-token"] = "VALUE_OF_ct0_COOKIE"

# --------------------------------------------------

# see transaction.py, line 125
time_now = math.floor((time.time() * 1000 - 1682924400 * 1000) / 1000)
print(f"Using time_now = {time_now}")

for random_num in (3, 79, 3, 51, 92, 79, 5):
    session.headers["x-client-transaction-id"] = ct.generate_transaction_id(method="GET", path=api_url_path,
                                                                            time_now=time_now, random_num=random_num)
    api = session.get(api_url)
    print(f"Status: {api.status_code} - random_num: {random_num: 3d} - TID: {session.headers['x-client-transaction-id']}")

It should print something like the following, showing that a 404 is returned when the TID is reused:

Using time_now = 63831002
Status: 200 - random_num:   3 - TID: **********************************************************************************************
Status: 200 - random_num:  79 - TID: **********************************************************************************************
Status: 404 - random_num:   3 - TID: **********************************************************************************************
Status: 200 - random_num:  51 - TID: **********************************************************************************************
Status: 200 - random_num:  92 - TID: **********************************************************************************************
Status: 404 - random_num:  79 - TID: **********************************************************************************************
Status: 200 - random_num:   5 - TID: **********************************************************************************************

You may notice that it fails on the first reuse of a random_num, which conflicts with what I had said earlier:

As a note: I think that I was always experiencing a 404 on the second request originally, but when testing right now, I'm getting it on the third.

I had made an observational mistake when I was testing at that time. The behavior show by this test script, where a TID cannot be reused once, is what what I was actually seeing.

[JiaoZaiShiJie]

@wyj8bc5odzfl Each generated id is used once, waited 15 minutes, and then unblocked

[JiaoZaiShiJie]

In addition, you can generate ids ahead of time and I'm sure they won't expire within 18 hours

[iSarabjitDhiman]

@wyj8bc5odzfl Each generated id is used once, waited 15 minutes, and then unblocked

Thats what I was saying that the ids are reusable, in fact people are already reusing these pre generated Ids.

Though the solution @wyj8bc5odzfl provided, makes the app more stable in those rare cases, but a while loop isn't a perfect solution. It will affect the speed, I would rather leave the Tid generation on the user side rather than handling it in a while loop. They can do a time sleep on their side if they are making requests in bulk or as @JiaoZaiShiJie suggested, generate these Ids beforehand by passing a future timestamp (time_now) in generate_transaction_id method.

I totally appreciate @wyj8bc5odzfl for providing a usable code snippet, it may come handy if anyone is trying the same approach.

[runetech0]

@wyj8bc5odzfl Each generated id is used once, waited 15 minutes, and then unblocked

I'm a bit confused. How would an ID work after 15 minutes when the timestamp used in that will be 15 minutes before using?

Edit: Also, what's the point of generating the Transaction IDs ahead of time when you can just save all the network values and then pass them anytime to generate a new Transaction ID and that just works? I mean really am I missing something here?

[wyj8bc5odzfl]

I'm a bit confused. How would an ID work after 15 minutes when the timestamp used in that will be 15 minutes before using?

The actual value of the timestamp doesn't appear to matter much - I just tried setting it to one week ago and it worked fine.

@wyj8bc5odzfl Each generated id is used once, waited 15 minutes, and then unblocked

Thats what I was saying that the ids are reusable, in fact people are already reusing these pre generated Ids.

I think I understand what you mean. I generate a new ID for each request in my own scripts, so having it generate a working ID every time is a priority for me, but I can see how building up a pool of IDs and rotating through them over time as they become blocked/unblocked would also be usable.

I'm fine with you deciding that the best approach here is up to the end user. Though, since it happens rarely and the cause is hard to notice unless you are carefully examining the ID that was used, I wonder if it would be worth making note of this behavior somewhere so that it doesn't surprise anyone.

I totally appreciate @wyj8bc5odzfl for providing a usable code snippet, it may come handy if anyone is trying the same approach.

Thanks! I have slightly changed the code I use since my pull request, so I'll share that below in case someone would like to use it. The caveats mentioned in the second half of my comment here apply to it.

--- a/x_client_transaction/transaction.py
+++ b/x_client_transaction/transaction.py
@@ -29,6 +29,7 @@ class ClientTransaction:
         self.key_bytes = self.get_key_bytes(key=self.key)
         self.animation_key = self.get_animation_key(
             key_bytes=self.key_bytes, home_page_response=self.home_page_response)
+        self.time_random_pairs = {(-1, -1)}
 
     def get_indices(self, ondemand_file_response: bs4.BeautifulSoup):
         key_byte_indices = []
@@ -122,8 +123,15 @@ class ClientTransaction:
         return animation_key
 
     def generate_transaction_id(self, method: str, path: str, home_page_response: Optional[bs4.BeautifulSoup] = None, key: Optional[str] = None, animation_key: Optional[str] = None, time_now: Optional[int] = None) -> str:
-        time_now = time_now or math.floor(
-            (time.time() * 1000 - 1682924400 * 1000) / 1000)
+        time_random_pair = (-1, -1)
+        while time_random_pair in self.time_random_pairs:
+            _time_now = time_now or math.floor(
+                (time.time() * 1000 - 1682924400 * 1000) / 1000)
+            random_num = random.randint(0, 255)
+            time_random_pair = (_time_now, random_num)
+        self.time_random_pairs.add(time_random_pair)
+        time_now = _time_now
+
         time_now_bytes = [(time_now >> (i * 8)) & 0xFF for i in range(4)]
         key = key or self.key or self.get_key(
             home_page_response=home_page_response)
@@ -135,7 +143,6 @@ class ClientTransaction:
             f"{method}!{path}!{time_now}{self.random_keyword}{animation_key}".encode()).digest()
         # hash_bytes = [int(hash_val[i]) for i in range(len(hash_val))]
         hash_bytes = list(hash_val)
-        random_num = random.randint(0, 255)
         bytes_arr = [*key_bytes, *time_now_bytes, *
                      hash_bytes[:16], self.random_number]
         out = bytearray(

[mikf]

Maybe this problem csn be solved, or at least alleviated, by using the fractional part of the time.time() result as random_num. That way it would only generate the same transaction ID in a 4ms window.

now = time.time()
timestamp = int(now)
time_now = timestamp - 1682924400
random_num = int((now - timestamp) * 256.0)

[wyj8bc5odzfl]

Maybe this problem csn be solved, or at least alleviated, by using the fractional part of the time.time() result as random_num. That way it would only generate the same transaction ID in a 4ms window.

I've thought about that, but was a bit weary that Twitter could easily detect the non-random, time-based pattern over a few requests and decide to ban or restrict accounts based on it.
I think it's good besides that risk, though!

(It wouldn't work well for the users who are generating a batch of IDs to use later, however.)

[mikf]

Maybe combine fractional part with a "real" random number then, e.g. 4 bit fractional + 4 bit random. These values' bits could even be interleaved to hide a time-based pattern even more, like frfrfrfr

[JiaoZaiShiJie]

每个生成的 id 使用一次，等待 15 分钟，然后解封

我有点困惑。如果 ID 使用的时间戳在使用前 15 分钟，则 ID 在 15 分钟后如何工作？

编辑：另外，当您可以保存所有网络值，然后随时传递它们以生成新的交易 ID 并且这就可以工作时，提前生成交易 ID 有什么意义呢？我的意思是，我真的在这里遗漏了什么吗？

I'm mostly concurrent so I need to test that the result is indeed if XID can be accessed in 15 minutes