Is there an extra x-xp-forwarded-for in the x protocol header?

[tuiterj]

Today I opened it and saw that there is now x-xp-forwarded-for which didn't seem to be there before

[c7a2d9e]

Non-required header.

[Mannumanman]

maybe required in the future

[tuiterj]

maybe required in the future

It should be needed soon. It has a time limit. It is valid within 5 minutes. It seems like this

[iSarabjitDhiman]

Thanks @tuiterj for pointing out, I will take a look soon.
Thats what I was thinking @Mannumanman
It's not required yet but it might be in future. They did the same with this X-Client-Transaction-Id header, first it was required only for some specific endpoints and look where we at now. It's used everywhere even guest users are no exception.
It seems we are early to the party and have enough time to do some digging and take care of it before its becomes a requirement.

Let's keep everyone posted, if anyone finds out anything, feel free to write it here. I will do my research in the meantime.

[tuiterj]

@iSarabjitDhiman This looks more complicated than X-Client-Transaction-Id, with 736 characters, and may also contain a server key verification signature. This was added not long ago, and it was not there before. It is not necessary now, but it will definitely be needed in the future, otherwise Twitter would not do it! Twitter has made another big move in anti-scraping and anti-automation, and it is always upgrading!

[dangeredwolf]

Looking at it so far

https://abs.twimg.com/responsive-web/client-web/loader.FwdForSdk-594367b3.f0d8f13a.js
https://abs.twimg.com/responsive-web/client-web/loader.FwdForSdk-ae793f77.7aea6cda.js

• It's generated using a WASM blob and other JavaScript functions
• It's only generated once and reused across requests (not sure for how long)

[tuiterj]

@iSarabjitDhiman After updating the x-xp-forwarded-for header, the original X-Client-UUID disappeared. Was it merged into x-xp-forwarded-for?

[fa0311]

I made this with Node.js
https://github.com/fa0311/x-xp-forwarded-for-generater

[iSarabjitDhiman]

I made this with Node.js
https://github.com/fa0311/x-xp-forwarded-for-generater

Thank you @fa0311
Really appreciate it. I will take a look shortly and see if it can be ported to python.

Thanks

[heysurfer]

const std::string base_data = R"({
        "webgl_fingerprint":"",
        "canvas_fingerprint":"",
        "navigator_properties":{
            "hasBeenActive":"true",
            "userAgent":"",
            "webdriver":"false"
        },
        "codec_fingerprint":"",
        "audio_fingerprint":"",
        "audio_properties":null,
        "created_at":0
    })";

std::vector<unsigned char> tag(TAG_SIZE);
if (EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_GET_TAG, TAG_SIZE, tag.data()) != 1) {
	EVP_CIPHER_CTX_free(ctx);
	throw std::runtime_error("Failed to get tag");
}

EVP_CIPHER_CTX_free(ctx);

std::vector<unsigned char> result;
result.insert(result.end(), nonce.begin(), nonce.end());
result.insert(result.end(), ciphertext.begin(), ciphertext.begin() + ciphertext_len);
result.insert(result.end(), tag.begin(), tag.end());

return bytes_to_hex(result);

It uses classic AES encryption. It first combines the main key with the guest ID, hashes this using SHA-256, and uses the result as the AES key. Then, it generates a 12-byte random nonce. After that, it encrypts using AES-GCM and obtains a tag for the encrypted data

Finally, it concatenates all this data and converts it to hex: nonce + encrypted text + tag
There is no fingerprinting or canvas information—only created_at and webdriver

[iSarabjitDhiman]

If anyone is interested, this guy seems to have solved this x-xp-forwarded-for header thing.

https://github.com/dsekz/twitter-x-xp-forwarded-for-header

Its almost as same as @heysurfer explained, but just written in python and maybe with some additional information.