Events stop firing

Dec 3, 2014 at 10:10 AM
Edited Dec 3, 2014 at 10:31 AM
We are running TweetInvi as a server process which collects Tweets and Direct Messages for a specific account using filterstream and userstream.

Most exercising of the code has been done following starting the server code. We are now doing longer runs and it appears that if the server is left running for a period, say overnight, the next day it is not possible to trigger any new events.

Is there anything in Twitters service which limits how long Tweetinvi can maintain a connection, or is there some other way of working around this.

Thanks

[EDIT} In the latest cases of this, it looks like a window of around 16 hours was left from the last event firing correctly, until generating a tweet and a DM which did not generate expected events.
Dec 4, 2014 at 6:56 PM
Edited Dec 4, 2014 at 6:57 PM
Hi,

You are not the first one reporting issues with the Stream. It looks like Twitter is having issues.
I will have to update the code to handle the disconnect. Here is a temporary fix for you.
var s = Stream.CreateSampleStream();
s.TweetReceived += (sender, args) => { Console.WriteLine(args.Tweet);}

// As per twitter documentation, the stream should be reseted after 30 seconds if nothing has been received by the WebRequest
var resetTimer = new System.Timers.Timer(30000);
resetTimer.Elapsed += (o, eventArgs) =>
{
    s.StopStream();
    s.StartStream();
};

s.JsonObjectReceived += (sender, args) =>
{
    resetTimer.Stop();
    resetTimer.Start();
};

s.StartStream();
This code is similar to what I will implement in coming versions.
Linvi
Marked as answer by linvi on 12/10/2014 at 1:35 AM
Dec 5, 2014 at 9:35 AM
Thanks for the reply.

Isnt a 90s timer needed for stalls ?

https://dev.twitter.com/streaming/overview/connecting
Dec 8, 2014 at 6:19 PM
Yes sorry, I remembered 30 seconds, but this is the stalls.
90 seconds it is.

Please let me know if this solves your problem.

Linvi
Dec 9, 2014 at 11:44 AM
Hi Linvi

It seems to work OK, but I don't know if this is because the problem is intermittent or not.

Having said that, it is exactly how I would have done it, and I think it does what it is supposed to.

On the down side, do you think that reconnecting every 90s or so is likely to attract rate limiting of some kind ?

I would prefer it if this code was in a library so it was not being done at two levels - do you have any date planned for the next library version which will contain this.

Many thanks
MartR
Dec 9, 2014 at 6:20 PM
Hi,

Thank you for the feedback.

On the down side, do you think that reconnecting every 90s or so is likely to attract rate limiting of some kind ?

I have not tested it but I believe that stopping and starting the timer reset it.
Therefore the IntervalElapsed event should only be raised once after 90 seconds.

I would prefer it if this code was in a library so it was not being done at two levels - do you have any date planned for the next library version which will contain this.

Yes I know, I have loads of things to do on Tweetinvi, and unfortunately, I have very little time left at the current time to work on it. Just enough to help people on the forum.

I will soon have some Christmast holidays. I expect I will catch up with the development during the holidays. So I would say you should have something available around the 20th of December.

Linvi
Jan 2, 2015 at 3:53 PM
Hi Linvi.
Could I ask if there is any progress made on this yet. Do you have a planned date for the next library ?
Thanks
MartR
Jan 12, 2015 at 10:38 AM
HI Linvi

I would be very grateful to hear of planned dates for a library fix for this.

Thanks
MartR
Feb 3, 2015 at 5:03 PM
Hi Linvi any news ?

Thanks
MartR
Feb 5, 2015 at 11:16 AM
Edited Feb 5, 2015 at 4:34 PM
In the meantime I have added this to my application code as a work around.

Although using a timer to do :

StopStream
StartStream
works OK for a user stream

calling
StopStream
StartStreamMatchingAnyCondition

does NOT restart a filtered stream which was running fine before the timer stopped.

Any idea why.
This could be a showstopper.

EDIT it looks as if this is related to a race condition between the worker thread and the timer thread. For some reason I amd getting two stopstreams generated sometimes, the second AFTER I have just restarted the stream.
Feb 6, 2015 at 10:55 AM
As a followup to the above problem

Calling
stopstream
startstream

on the filter stream in the timer thread is fine if you are not receiving matching tweets at the time.

If you are receiving matching tweets at the time, the effect is that an extra Stopstream is generated After the start stream. i.e. you will see the events firing like this :

StopStream
StartStream
StopStream

Which means the attempt to recycle (during data) leaves the collection stalled - which is ironic as this is what we are trying to avoid by recycling the connection.

I dont know where the second StopStream is coming from, it has no call stack when observed in the debugger which suggests it is being called from the TweetInvi library perhaps ?

It is not being called from the MatchingTweet event as I have removed all calls to StopStream except the one in the timer.

This is nasty, I may have to try and pause the stream first and then delay before stopping and starting it. Either way it is not nice.

I would really appreciate some input on this as this problem has been open for some weeks now.
Feb 9, 2015 at 5:39 PM
Hi MartR,

I believe you noticed I am not around that much. I am sorry for that I am working on 3 projects at the same time and got very little time to help you.
The stream fix will be the first I release as soon as I can find a safe solution to the problem.

Sorry for the big delays.

Linvi
Feb 9, 2015 at 6:32 PM
Please try to get latest from Source Code (Tweetinvi 0.9.5.x).
I applied a fix that should resolve your problem.

When a stream Timeout, the StreamStopped event is raised. The disonnect message has a code of 503 and a description of timeout.

Cheers,
Linvi
Feb 10, 2015 at 9:15 AM
Thanks for getting back to me. I have now implemented this in my application anyway. The problem I was having (stopstream being called after startstream) was due to me setting a timeout of 30s so that I could soak test with no twitter events arriving. It looks like the error was being cause by rate limiting. When I move the timeout out to 90s as per the Twitter documentation, it seems fine.
Feb 10, 2015 at 1:34 PM
Edited Feb 10, 2015 at 1:35 PM
linvi wrote:
Please try to get latest from Source Code (Tweetinvi 0.9.5.x).
I applied a fix that should resolve your problem.

When a stream Timeout, the StreamStopped event is raised. The disonnect message has a code of 503 and a description of timeout.

Cheers,
Linvi
Also, please note that there is a LimitReached event on the stream. I never got this to fire despite believing that I was being rate limited. In fact the symptoms were that on calling Startstream, exceptions were thown (followed by a stopstream event) but GetLastException did not contain anything - so it is difficult to detect when the stall limit is set too low because no errors seem to be produced. (Using 0.9.4.x)
Feb 10, 2015 at 6:18 PM
Tweetinvi already implement the LimitReached event:
var s = Stream.CreateFilteredStream();
s.LimitReached += (sender, args) =>
{
    var missedTweets = args.NumberOfTweetsNotReceived;
};
Receiving this event do not mean that you is not related with Rate Limits but it means that you are trying to get more than 1% of all the Tweets written on Twitter.
For example if you create a Filter stream with lots of famous keywords, you can receive this event.
Tweetinvi will let you know the number of tweets that you missed but obviously cannot give you the missing tweets.

Linvi
Feb 16, 2015 at 9:35 AM
On the general subject of dealing with stalls, there is still an issue with the proposed solution.

Even at 90s, for some reason Twitter stops the stream occasionally before the 90 timeout has had a chance to recycle.
No idea why it does this, but I did a 65 hour soak test, recycling a user stream and a filter stream every 90 seconds.

That is approximately 2600 restarts of each stream. However the code detected that each stream was stopped on around 60 occasions before the timeout had elapsed (specifically, a stop stream was fired by twitter, where the duration of the run was less than 80 seconds).

The problem with this is that the code needs to be able to attempt to restart immediately rather than just wait for the next timer to elapse - otherwise it might miss events. Not only that, according to the documentation, it then needs to back off exponentially with any necessary retries if that connection is not successful for some reason.

I think this logic needs to be added to the next version of the library.

Let me know what you think.
Thanks
Feb 17, 2015 at 10:35 AM
Hi,

I don't know if this could help, but I have the same issues with my code.
Nevertheless, I noticed something very strange while testing : my problems are related to the machine I test my code on.
I have 6 machines : 2 windows 2012 R2 Datacenter, 1 windows 2008 R2 Enterprise, 1 windows 2008 R2 standard, 1 ubuntu 14.04 (running different VMs) and 1 windows 8.1.

Thing is, the "stall" effect occurs only on my 2 machines running windows 2012 R2 Datacenter.

As all machines are physically in different places, I already excluded physical network issues by different tests and analysis (for instance, the ubuntu machine is on the same network as the machines causing problems).
I also tested the streaming API using curl, and I have issues on the same machines (curl closing connection with a "transfer closed with outstanding read data remaining" message).

I'm starting to run out of hypothesis there... Could it be a default network configuration issue like a limit on tcp packets or a too short timeout ?
Feb 20, 2015 at 11:10 AM
Edited Feb 20, 2015 at 11:12 AM
Hi again,

I think I will release 0.9.5.0 with the current behavior because it is long awaited and because I will need more details from you.
Would you mind if we continue positing on this thread as long as I have questions on the subject?

I have also created a Work Item called "Stream Reset Improvements" (https://tweetinvi.codeplex.com/workitem/2513).

Thank you for your help.
Linvi
Mar 20, 2015 at 6:34 PM
Edited Mar 20, 2015 at 6:34 PM
Hi MartR,

I am going to work on improvements on the stream and stalls. But in order to do this I need some explanation from both of you.

Even at 90s, for some reason Twitter stops the stream occasionally before the 90 timeout has had a chance to recycle.

From what I understand you have a stream that stops working before the 90 first seconds it started? I am not sure I understand this clearly.

The problem with this is that the code needs to be able to attempt to restart immediately rather than just wait for the next timer to elapse - otherwise it might miss events.

I agree but I cannot really see a solution to this. Twitter clearly states that a stream we can know that a stream is dead after 90 seconds without any information.
My worry is that if we specify a limit at 70 seconds for example, and the stream is not dead. At second 71 we are restarting the stream and a Tweet is added to the stream. We therefore missed the event. When the stream has restarted at second 72 we have missed it without knowing.

Do you think it would be a good idea to give the option to developers to themselves specify a timeout after which the stream will restart?

Not only that, according to the documentation, it then needs to back off exponentially with any necessary retries if that connection is not successful for some reason.

The issue with the back off is that it is only available to user with elevated privileges on the Stream API. Do you think it is worth adding such a feature as so few developers will benefit from it?

Regards,
Linvi
Mar 20, 2015 at 6:38 PM
Hi StfSki,

I am going to work on improvements on the stream and stalls. But in order to do this I need some explanation from both of you.

Thing is, the "stall" effect occurs only on my 2 machines running windows 2012 R2 Datacenter.

In technical details, what do you call the "stall" effect?

Cheers,
Linvi
Mar 20, 2015 at 6:51 PM
Hi Linvi,

What I meant is, the frequency of received messages slows down to finally reach zero without receiving any errors.

Stéphane Sochacki

Le 20 mars 2015 à 18:38, linvi <[email removed]> a écrit :

From: linvi

Hi StfSki,

I am going to work on improvements on the stream and stalls. But in order to do this I need some explanation from both of you.

Thing is, the "stall" effect occurs only on my 2 machines running windows 2012 R2 Datacenter.

In technical details, what do you call the "stall" effect?

Cheers,
Linvi
Mar 24, 2015 at 1:01 PM
Edited Mar 24, 2015 at 1:02 PM
Hi Linvi, comments at the bottom. Thanks

1.
Even at 90s, for some reason Twitter stops the stream occasionally before the 90 timeout has had a chance to recycle.

From what I understand you have a stream that stops working before the 90 first seconds it started? I am not sure I understand this clearly.

2.
The problem with this is that the code needs to be able to attempt to restart immediately rather than just wait for the next timer to elapse - otherwise it might miss events.

I agree but I cannot really see a solution to this. Twitter clearly states that a stream we can know that a stream is dead after 90 seconds without any information.
My worry is that if we specify a limit at 70 seconds for example, and the stream is not dead. At second 71 we are restarting the stream and a Tweet is added to the stream. We therefore missed the event. When the stream has restarted at second 72 we have missed it without knowing.

Do you think it would be a good idea to give the option to developers to themselves specify a timeout after which the stream will restart?

3.
Not only that, according to the documentation, it then needs to back off exponentially with any necessary retries if that connection is not successful for some reason.

The issue with the back off is that it is only available to user with elevated privileges on the Stream API. Do you think it is worth adding such a feature as so few developers will benefit from it?






(1). I soak tested for 24 hours. I had logging which logs started and stop events. I also made a note of the time that the start event began and checked it in the stop event. This allowed me to check for short sessions.

What I saw in normal use was a startsession event logged, followed 90s later by a stopsession event logged. In some cases though the stopsession fired before the 90 timer had fired which made me conclude that this was something else in the twitter API which was stopping the session early. The problem I had of course without adding more code to work around that was that the "anti stall" code was actually stalling and remained stalled until the rest of the 90s window expired to restart the session. Not nice. We are throwing away data.

(2). No I think you are right to enforce what the docs say. And if they say 90s then that should be it. The only problem is if the docs change at a later date then a method to override might be useful.

(3). I agree. It is not worth doing.
Mar 24, 2015 at 2:33 PM
Hi all,
I have a question about the initially proposed fix for this (using a timer that gets reset on JsonObjectReceived).
In the Twitter docs for this (https://dev.twitter.com/streaming/overview/connecting section "Stalls") it says that the newline character is used as the keep-alive. This won't be valid JSON; so will it trigger the JsonObjectReceived event?
Thanks,
Josh
Mar 24, 2015 at 11:12 PM
@StfSki

I think you are probably experiencing the Stalls of Twitter Stream API which is a known and documented behavior.
This stalls happens when your code is not going fast enough to handle the Tweets it receives from Twitter.

This can happen for example when you need to store data in a database or when you need to send these information back on the network.
Basically Twitter waits for you to get back on track but if you can't it sends you a Warning message to let you know that your are falling behind.
When the falling behind message is at 100% the stream connection is closed by Twitter.
var s = Stream.CreateSampleStream();
s.WarningFallingBehindDetected += (sender, args) =>
{
    Console.WriteLine(args.WarningMessage.PercentFull);
};
Please have a look into this.
Cheers,
Linvi
Mar 24, 2015 at 11:15 PM
@JKeegan

The JsonObjectReceived is not really a Json Object. Instead it should be called MessageReceived.
So any string received within the stream is analyzed by Tweetinvi.

In any case I am waiting for the delay or when a response is received from the Stream.
// Code of Tweetinvi awaiting for message
var requestTask = _currentReader.ReadLineAsync();
var resultingTask = TaskEx.WhenAny(requestTask, TaskEx.Delay(STREAM_DISCONNECTED_DELAY)).Result;
Cheers,
Linvi
Mar 24, 2015 at 11:23 PM
@MartR

Thanks for your response. But please let me clarify to ensure that I understand clearly.

I soak tested for 24 hours. I had logging which logs started and stop events. I also made a note of the time that the start event began and checked it in the stop event. This allowed me to check for short sessions.

You did not intentionally created short sessions, you just listened to the events in order have a good overview of the disconnect.

What I saw in normal use was a startsession event logged, followed 90s later by a stopsession event logged.

Again this was not intentional, you were disconnected from the stream by Twitter?

In some cases though the stopsession fired before the 90 timer had fired which made me conclude that this was something else in the twitter API which was stopping the session early.

Can you have a look 2 messages above, the reply to StfSki. Do you think it might be what you experienced?

The problem I had of course without adding more code to work around that was that the "anti stall" code was actually stalling and remained stalled until the rest of the 90s window expired to restart the session. Not nice. We are throwing away data.

Here I think I got lost, but from what I understand, the code that you created to ensure that your stream was kept alive was causing it to stall.

Final Question

What do you think I could do to improve Tweetinvi and help you with these issues?

Thanks for the help and this detailed answer.
Mar 25, 2015 at 9:43 AM
Eureka !

I think I fixed it, I'll try to explain clearly.

Symptoms :
  • Create a stream
  • Start listening
  • Messages start to arrive
  • Slowly, the number of received messages from Twitter (even the linefeeds which are supposed to keep the connection alive) decreases
  • Finally, nothing comes back from Twitter
Diagnostic :
  • Once your stream is created, be sure to catch all possible events from Twitter (for debugging purpose, use a filter you'll be sure to catch a lot of data, i.e. track=twitter)
  • Applies @linvi's very elegant solution with the timer to be sure your code is as robust as possible
  • Try, if you can, to execute your code on many different machines, different configurations (OS, network, provider etc.)
  • Grab the Windows version of curl (with SSH), and use it to test your stream outside of your code (I use Twitter oAuth tool to generate curl request, check at the end of this page https://dev.twitter.com/streaming/reference/post/statuses/filter for instance).
  • Check Twitter API Status (but what are the odds the problem could come from there ?)
If this "stall" effect still occurs, you'll have curl errors like "transfer closed with outstanding read data remaining".
At this point, you can rule out any software issues and start searching at a lower level.

After a little googling, I found this page : https://technet.microsoft.com/en-us/library/dd349797%28v=ws.10%29.aspx.
WARNING : always backup your registry before playing with it !

I checked my registry, none of those values were event there, so I added them and... everything started working perfectly and immediately.

I'm convinced that, there is absolutely no software explanation for not receiving any data from Twitter, the documentation is clear, they'll always, at least, send a linefeed every 30 seconds.
Thus, if you stop receiving data without catching any exception (missing, falling behind, quotas etc.), the problem must come from somewhere else at a lower level (TCP, network, hardware etc.).

I really hope this could help and save time.
Mar 25, 2015 at 10:43 AM
Edited Mar 25, 2015 at 10:44 AM
Thanks for your response. But please let me clarify to ensure that I understand clearly.

1)
I soak tested for 24 hours. I had logging which logs started and stop events. I also made a note of the time that the start event began and checked it in the stop event. This allowed me to check for short sessions.

You did not intentionally created short sessions, you just listened to the events in order have a good overview of the disconnect.

2)
What I saw in normal use was a startsession event logged, followed 90s later by a stopsession event logged.

Again this was not intentional, you were disconnected from the stream by Twitter?

3)
In some cases though the stopsession fired before the 90 timer had fired which made me conclude that this was something else in the twitter API which was stopping the session early.

Can you have a look 2 messages above, the reply to StfSki. Do you think it might be what you experienced?

4)
The problem I had of course without adding more code to work around that was that the "anti stall" code was actually stalling and remained stalled until the rest of the 90s window expired to restart the session. Not nice. We are throwing away data.

Here I think I got lost, but from what I understand, the code that you created to ensure that your stream was kept alive was causing it to stall.

Final Question
5)
What do you think I could do to improve Tweetinvi and help you with these issues?

Thanks for the help and this detailed answer.


1) No I wanted to keep the sessions open always, the only closing I did was when the 90 timer elapsed. So something else in Twitter was closing the session. There was virtually no data expected based on the filter, so the cause was not that there was too much data, and I was falling behind. Something else was causing twitter to close it. I have most if not all the events implemented with logging and I could see no other problems being reported.

2) Yes Twitter unexepectedly disconnected. On the soak test there were around 50 short sessions detected, some as short as 10s. This represented about 6% of the 90s sessions over that period.

3) No. I dont believe that was the cause. See answer 1,

4) No. I dont think my code was causing it to stall. Events for a normal session were as follows

A. Stream Start
B. 90 s timer fires and stops and restarts stream
C. Stream Stop
D. Stream Start

Events for a problem session were

A. Stream Start
X. Stream Stop (unexpected)
B. 90 s timer fires and stops and restarts stream
C. Stream Stop
D. Stream Start

So what I am saying that for the window between X and B there was no session open. And data which would have appeared would have been thrown away.

5) To have the Tweetinvi layer do the job of maintaining the session up all the time, whatever it takes, so that the application does not necessarily need to worry about it. If there are any occasions when then cant happen then better error reporting / event firing would be good (Note: I if this detail is already these, sorry but I have missed it).

Thanks !
Mar 25, 2015 at 11:22 AM
@StfSki

Thank you very much for this input. Do you have any idea which specific registry key change it from failing to working?

Linvi
Mar 25, 2015 at 11:38 AM
Hi Linvi,
Is that code in the current NuGet package?
What happens when the task returned is the delay? Does the stream close or fire some event that we must listen for?

Am I right in thinking that there is now no advantage whatsoever to having the original proposed fix?
Thanks,
Josh
Mar 25, 2015 at 11:41 AM
linvi wrote:
@StfSki

Thank you very much for this input. Do you have any idea which specific registry key change it from failing to working?

Linvi
To be completely honest, I didn't try each of the four keys independently.
However, using my intuition, I'd tell it's something about the KeepAliveTime (but as the PerformRouterDiscovery key is about the IRDP protocol, perhaps this one is important as well).
Mar 25, 2015 at 11:42 AM
@MartR

Thanks, I understand perfectly what you were trying to do now.
I have made multiple tests and never got such behavior on my 2 machines. I am not sure what could be causing such behavior on yours.

Could you please update your code and add the following lines:
var fs = Stream.CreateFilteredStream();
fs.JsonObjectReceived += (sender, args) =>
{
    if (!string.IsNullOrWhiteSpace(args.Json))
    {
        Console.WriteLine(args.Json);
    }
};

fs.StreamStopped += (sender, args) =>
{
    var lastException = ExceptionHandler.GetLastException();
    if (lastException != null)
    {
        Console.WriteLine("An error occurred");
        ExceptionHandler.ClearLoggedExceptions();
    }

    Console.WriteLine("Stream Stopped at " + DateTime.Now.ToLongTimeString());
};
Please let me know if an Exception is thrown and which error it was?
If no exception you might receive a Json message from Twitter just before being disconnected. Could you please let me know which message you received if any?

Cheers,
Linvi
Mar 25, 2015 at 3:18 PM
@MartR

I have just had this in my mind. The ExceptionHandler is unique per Thread.
But the Stream are executed in a different thread and therefore when you will invoke the ExceptionHandler it will never contain any error.

In the meantime please deactivate the SwallowWebExceptions:
ExceptionHandler.SwallowWebExceptions = false;
You will receive AggregateException if any. At this point look into the InnerException and you will probably get a WebException or TwitterException.

Cheers,
Linvi

PS : I will work on improvements to manage Exceptions in streams