Do We Need Site Streams ?

Sep 15, 2014 at 10:48 AM
Edited Sep 15, 2014 at 10:48 AM
Here is my case

A web App accept many users and this App listen to each user events as it happen . when i said many users i mean 20k twitter user ,

with user stream i got this work but only for a single user , i have been told in order to allowed multiple users use this app you need Site stream > is that's true and if it does is there is any workaround this ? twitter stop giving site stream access so that not an Option
Sep 15, 2014 at 12:36 PM
Edited Sep 15, 2014 at 12:37 PM
i also know for fact that userstream can work with multiple users if each user have a different IP then the others , for example i have an IOS App that uses userstream and it used by tuns of people with no problem ,

so i wonder if we created some proxy platform at the server side for this web App and assign an IP for each user if that will work ?
Coordinator
Sep 15, 2014 at 10:51 PM
Edited Sep 16, 2014 at 7:58 AM
Hello there,

SiteStream are very good but even after requesting an access 3 times, I never got one, so I am not sure how good it it.
It is possible to have multiple UserStream running on the same IP but they are limited and the only alternative you have is to use multiple OAuth application credentials (which is sometime the only solution to multiple rate-limits).
// Create the credentials, this is where you might want to use multiple application tokens.
var cred1 = TwitterCredentials.CreateCredentials("Access_Token", "Access_Token_Secret", "Consumer_Key", "Consumer_Secret");
var cred2 = TwitterCredentials.CreateCredentials("Access_Token", "Access_Token_Secret", "Consumer_Key", "Consumer_Secret");

// Create the streams
var us1 = Stream.CreateUserStream();
var us2 = Stream.CreateUserStream();

// Register the events you want
us1.JsonObjectReceived += (sender, args) =>
{
    Console.WriteLine("user1 received message");
};

us2.JsonObjectReceived += (sender, args) =>
{
    Console.WriteLine("user2 received message");
};

// Start the streams
TwitterCredentials.ExecuteOperationWithCredentials(cred1, () => us1.StartStreamAsync());
TwitterCredentials.ExecuteOperationWithCredentials(cred2, () => us2.StartStreamAsync());
Please have a careful read to this documentation page to understand the limitation Twitter API uses on UserStream per application.

Please let me know if I can help you more.
Linvi
Sep 16, 2014 at 6:13 AM
thanks linvi , can multiple OAuth application credentials handle something like 10k userstream connection at the same time ? I'm expecting at least 10k user to be used my App so i wonder
Editor
Sep 16, 2014 at 6:20 AM
Sep 16, 2014 at 12:43 PM
@imlokesh
you mention that you did used multiple OAuth credentials , may i ask if you had any problem and how many users using your app ?
Editor
Sep 16, 2014 at 1:47 PM
Hi muts,

I do use tons of OAuth credentials in a single application but I don't use twitter streams at that level.

I use FilteredStream and I can generally open only two or three connections to the filtered stream endpoint from the same IP address. The official documentation says that you're allowed "only a handful" of streaming connections per IP address.

So I'm pretty sure you won't be able to handle 10k without proxies. Even with proxies, I don't think getting thousands of proxies will be really feasible for you unless you have a good budget. Public proxies won't cut it for streaming connections.

imlokesh
Coordinator
Sep 16, 2014 at 3:02 PM
To complete the answer of imlokesh, here is the description of Twitter concerning the UserStream.

Minimize the number of connections your application makes to User Streams. Each Twitter account is limited to only a few simultaneous User Streams connections per OAuth application, regardless of IP. Once the per-application limit is exceeded, the oldest connection will be terminated. An account logging in from too many instances of the same OAuth application will cycle connections as the application instances reconnect and disconnect each other.

So you can have multiple UserStream running for one application but their number will be adding up regardless of their ip.
Therefore you will not have any choice other than having multiple applications.

Please both of you note that this is not "authorized" by Twitter and this can result in having your applications shutdown (which happened to me once 2 years ago).

Linvi
Editor
Sep 16, 2014 at 4:48 PM
Edited Sep 16, 2014 at 4:49 PM
Hi Linvi,

I would interpret this slightly differently.

Each Twitter account is limited to only a few simultaneous User Streams connections per OAuth application, regardless of IP.

I think this limit is per account and not per application. Meaning, you being the user, you cannot have the same application run several streams for you.

From my experience, this limit is basically one! In my application, I allow the user to to perform some actions over a streaming connection. If I login to my application from two instances of the app, even on two different machines (read different ips), the older stream connection gets disconnected.

On the other hand, my application has hundreds of users. And I have never received complaints of a user not being able to connect to the api. This would not be possible if my application was limited to only a few simultaneous User Streams.

I think many twitter apps that allow full account management rely on User Streams. These applications can have thousands of users.

But you're right in that connecting to multiple streams using proxies may not be appreciated by twitter. But having proxy support in tweetinvi could have other valid uses too. :)

imlokesh
Coordinator
Sep 16, 2014 at 4:53 PM
I am currently working on it ;)
Sep 16, 2014 at 5:17 PM
We will try multiple OAuth application without proxies and see how that goes and how many users can handle ,
Sep 16, 2014 at 7:09 PM
Edited Sep 16, 2014 at 7:11 PM
@linvi
you give code example on how to use multiple OAuth application credentials but how you will do that automatic for example i want user to visit my web page and click on "authorize my account" button > goes to twitter then back to my web page

this will help a lot