Can't get more than two multi-threaded filter streams to run

Sep 2, 2015 at 10:16 PM
I am building an app that takes saved user OAuth tokens and opens a filtered stream for each one. Eventually it will save the stream somewhere but for now I am just writing it to the console for debugging.

The issue I have is that I am only able to get two streams to return data. Which stream gets to output seems to be random but it is always only two. I am connecting to Twitter using OAuth tokens from different accounts so I shouldn't be getting limited that way if I understand the documentation correctly.

I'm not sure what could be the issue and have been stuck at this problem for some time.

Here is a sample of the code I am running.
public class Program
{
    readonly static string CONSUMER_KEY = "";
    readonly static string CONSUMER_SECRET = "";

    public static void Main(string[] args)
    {
        // Get all Twitter authentications.
        var twitterAuths = new List<TwitterAuth>();
        using (PosterBoardContext context = new PosterBoardContext())
        {
            twitterAuths = context.TwitterAuths.ToList();
        }

        // Start a thread for each Twitter authentication we have.
        Console.WriteLine("Reading Streams for {0} users", twitterAuths.Count);
        foreach (TwitterAuth twitterAuth in twitterAuths)
        {
            Task.Factory.StartNew(() =>
            {
                // Set our user credentials.
                Auth.SetUserCredentials(CONSUMER_KEY, CONSUMER_SECRET, twitterAuth.AccessToken, twitterAuth.AccessTokenSecret);

                // Thread name for debugging and tracking.
                var threadName = "Thread " + twitterAuth.Id;

                // Create the filtered stream.
                var stream = Stream.CreateFilteredStream();

                // Add a track to follow, use something that will get a lot of tweets.
                stream.AddTrack("hello");

                // Write to the console when the stream starts or stops and when we get a matching tweet.
                stream.MatchingTweetReceived += (s, a) => { Console.WriteLine("{0} - {1}", threadName, a.Tweet.Text); };
                stream.StreamStarted += (s, a) => { Console.WriteLine("{0} - Stream Started", threadName); };
                stream.StreamStopped += (s, a) => { Console.WriteLine("{0} - Stream Stopped", threadName); };

                // Start the stream.
                stream.StartStreamMatchingAllConditions();
            });
        }

        Console.ReadKey();
    }
}
Sep 3, 2015 at 12:04 PM
Edited Sep 3, 2015 at 12:05 PM
Hello,

I do not think the issue comes from Tweetinvi but it is more likely that Twitter restricts the usage of multiple public streams with the same IP.
The thing is that this behavior was mentioned in their documentation, but it is no longer there.

The only statement I have been able to find in their new documentation is the following :

Connections
Each account may create only one standing connection to the public endpoints, and connecting to a public stream more than once with the same account credentials will cause the oldest connection to be disconnected.
Clients which make excessive connection attempts (both successful and unsuccessful) run the risk of having their IP automatically banned.

Source

From this statement your code should work (but we have to remember that the Twitter doc misses lots of information), so I am going to investigate this.

So before we even start could you change your StreamStopped event and let me know what you information you get from this.
stream.StreamStopped += (sender, args) =>
{
    if (args.Exception != null)
    {
        Console.WriteLine(args.Exception);
    }

    if (args.DisconnectMessage != null)
    {
        Console.WriteLine(args.DisconnectMessage);
    }
};
Cheers,
Linvi
Sep 3, 2015 at 4:27 PM
Linvi,

Thanks for the quick response. Your activity on the forums is super helpful.

I updated the code and ran it with your changes but the StreamStopped event never fires.

I would also note that I tested this with UserStreams and I get the same limit. Only two of the streams (random every time) get stream data.

Reading the documentation further I noticed that Twitter says connecting to a public stream more than once with the same account credentials will cause the oldest connection to be disconnected. If that is the case I should be fine since I am connecting as a different account with each thread so I shouldn't be limited that way.

I did some experimenting and found that if I ran two instances of my console application that I could get four streams running at the same time (two random ones from each app). They were from four different user accounts and they were all streaming in real time.

Could I have stumbled on a bug in Tweetinvi? I am running 0.9.9.5 from NuGet. Thanks again for all your help.
Sep 3, 2015 at 5:28 PM
Edited Sep 3, 2015 at 5:34 PM
Well it looks like a bug with Tweetinvi then. I will investigate and get back to you.
In the meantime could you please upgrade to the latest version (0.9.9.6), there has been few changes to the webrequests.

I don't think this will fix the issue but it is worth a try.

Feel free to up vote the work item (https://tweetinvi.codeplex.com/workitem/2644) to get live notification about changes.

Cheers,
Linvi
Sep 4, 2015 at 10:02 AM
I have been able to reproduce your issue and I am going to work on this as a priority.
This will be fixed and should be released under Tweetinvi 0.9.9.7.

Cheers,
Linvi
Sep 4, 2015 at 2:04 PM
Hello,

I have been able to find a fix for the issue. Now it implies refactoring the StreamTask to use the HttpClient.
As you are probably aware, the PCL version of System.Net WebRequest are more limited than the one of the .NET platform.
HttpClient provides few improvements notably concerning multiple threads hosting continuous webrequests.

Therefore, I do not think this fix should be available as a minor version (0.9.9.7). Instead I think it should be available within the next release of Tweetinvi.
That being said, I plan to release version 0.9.10.0 in 1/1.5 month.

Would you be happy to use the Source Code and compile it until the next release of the library?
If needed I can also build a .zip with all the required binaries.

Thank you for understanding.
Please let me know what you think.

Linvi