REST API Timeout. Cannot see requests getting sent

Apr 13, 2015 at 2:25 PM
Hi Linvi,
I have a program that polls the Twitter REST API for a number of Twitter Users.

I've been getting the following occasionally:
--- Date : 13/04/2015 14:21:48
URL : https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=JeremyClarkson&include_rts=True&exclude_replies=False&contributor_details=False&count=200&trim_user=False&include_entities=True
Code : 408
Error documentation description : Twitter was not able to perform your query within the Timeout limit of 10000 ms.

The program would run without error for some time (up to a week) and then queries would start timing out. Once one had timed out, more & more will time out until no queries are returning within 10s.

Restarting the program fixes the problem. You can even leave the broken instance still running and start a new instance of the program which will work fine.

I don't believe this to be network related as I have reproduced the error on a Windows Azure instance (which is completely separate from our own hardware that we usually run the application on).

I don't think the queries are even being sent to Twitter because when I leave TCPView open, once the timeouts are happening for all requests, the packet counts & sent and received bytes for each open connection doesn't change.

Up until the end of last week I was running it with about 5 users, but it now has about 40 users. Since I have added the additional users, these errors have started happening far more frequently. Now I'm lucky if it works for 5 minutes!

I've put together a small program to reproduce the problem which you can download at: https://www.dropbox.com/s/3m27jjco0e7ye5j/ReproduceBug130.zip?dl=0

You'll need 10+ Twitter users to be able to reproduce the error in a reasonable amount of time (usually under 5 mins).
This is currently a major problem for me, so I'd be very grateful if you could be of assistance.
Thanks,
Josh Keegan
Coordinator
Apr 13, 2015 at 3:44 PM
Hi,

This issue worries me. Could you please try to deactivate the Timeout and let me know what happens?
TweetinviConfig.APPLICATION_WEB_REQUEST_TIMEOUT = 0;
Thanks,
Linvi
Apr 13, 2015 at 4:08 PM
Hi Linvi,
Thanks for the quick response!

Changing TweetinviConfig.APPLICATION_WEB_REQUEST_TIMEOUT doesn't seem to have any effect, it still errors and says 10000ms in the error message.
I tried the values: 0, -1, 15000.
Cheers,
Josh
Coordinator
Apr 13, 2015 at 4:23 PM
Edited Apr 13, 2015 at 4:24 PM
Thanks for the feedback, I have opened a new bug that will be fixed in 0.9.7.0 release which should come out this week.

https://tweetinvi.codeplex.com/workitem/2546

Cheers,
Linvi
Apr 13, 2015 at 4:28 PM
Thanks Linvi,
I'll happily test any potential fix before the next release if you need. Just post here & i'll try it.
Cheers,
Josh
Coordinator
Apr 13, 2015 at 6:25 PM
Sorry my bad could you try using the CurrentThread settings.
TweetinviConfig.CURRENT_WEB_REQUEST_TIMEOUT = 0;
Linvi
Apr 13, 2015 at 6:50 PM
Edited Apr 13, 2015 at 7:01 PM
I'm getting the same results again. Also tried with 0, -1 and 15000 (all showing a timeout of 10000ms in the exception).

I assumed that one was thread-specific and so put it in the setThreadCredentials method in the sample application attached.
Cheers,
Josh

Edit: Some errors are showing 10000ms but some are showing 15000ms.

Edit 2: When set to 0, all of the exceptions are for the rate limit status endpoint & 10000ms.
When I set it to 15000 there is a mixture of errors. The user timeline endpoint timeout exceptions now contain "15000 ms" and the rate limits endpoint timeouts are mixed, with some being for 10000 ms and others for 15000.
A log file when set to 15000: http://pastebin.com/cdpukGXK
Coordinator
Apr 13, 2015 at 11:19 PM
Hi,

Please give me a couple of days to work on this issue. I might contact you back with some request for details.

Cheers,
Linvi
Apr 16, 2015 at 9:21 AM
Hi Linvi,
How are you getting on with this?
Do you need want more info?
Thanks,
Josh
Coordinator
Apr 16, 2015 at 10:03 AM
Hi Josh,

I did not have time to work on this issue yet. Could you check the number of threads running in your application when you encounter this issue please?

I will let you know more when I have worked on this.

Linvi
Apr 16, 2015 at 11:56 AM
Hi Linvi,
Fair enough; do you have a rough idea of when you'll be getting around to looking at this?
I ask because if it'll be a while I can look into implementing some sort of work-around in the mean time.

In the attached program to reproduce the issue it sits at ~39 threads with 10 users and ~88 threads with 36 users.
On the live application that first experienced the issue, it currently sits at ~239 threads, with the same 36 users.
Cheers,
Josh
Coordinator
Apr 16, 2015 at 1:05 PM
Edited Apr 16, 2015 at 1:33 PM
I will try my best to help you as much as you can. Your bug is my priority.

IMPORTANT : Would you please let me know if you are using a proxy please?
239 threads seems quite high. Hopefully reading your code no deadlock should appear but please be quite careful with such a number of threads.

Linvi
Apr 16, 2015 at 1:34 PM
I'm not using a proxy (and haven't tested it with one).

200+ is a fair few threads, and that will continue to rise as more users are added & more functionality is implemented, but it's sitting pretty for now :)
At some point I'll probably have to switch away from having a thread per user, per API endpoint being polled, even if only to reduce the overheads involved.
As for deadlocks: currently I cannot think of any situation in the production code where a deadlock can occur (off of the top of my head), and lock-free methods/objects are used where possible (e.g. ConcurrentDictionary & use of Interlocked methods) but it will definitely need careful consideration in future as it grows.
Cheers,
Josh
Coordinator
Apr 16, 2015 at 1:43 PM
Could you please replace your WebLogic dll by the following:

https://www.codeplex.com/Download/AttachmentDownload.ashx?ProjectName=tweetinvi&WorkItemId=2546&FileAttachmentId=3379

Let me know if this fixes anything.

Linvi
Apr 16, 2015 at 1:58 PM
Hi Linvi,
That won't run:

Unhandled Exception: System.TypeInitializationException: The type initializer fo
r 'Tweetinvi.TweetinviConfig' threw an exception. ---> System.TypeInitialization
Exception: The type initializer for 'Tweetinvi.TweetinviContainer' threw an exce
ption. ---> System.IO.FileLoadException: Could not load file or assembly 'Tweeti
nvi.WebLogic, Version=0.9.6.1, Culture=neutral, PublicKeyToken=null' or one of i
ts dependencies. The located assembly's manifest definition does not match the a
ssembly reference. (Exception from HRESULT: 0x80131040)
at Tweetinvi.Injectinvi.AutofacContainer.RegisterModules()
at Tweetinvi.Injectinvi.AutofacContainer.Initialize()
at Tweetinvi.Injectinvi.AutofacContainer..ctor()
at Tweetinvi.TweetinviContainer..cctor()
--- End of inner exception stack trace ---
at Tweetinvi.TweetinviContainer.ResolveT
at Tweetinvi.TweetinviConfig..cctor()
--- End of inner exception stack trace ---
at Tweetinvi.TweetinviConfig.set_APPLICATION_WEB_REQUEST_TIMEOUT(Int32 value)

at ReproduceBug130.Program.run() in c:\Users\Josh Keegan\Documents\Visual Stu
dio 2013\Projects\ReproduceBug130\ReproduceBug130\Program.cs:line 132
at ReproduceBug130.Program.Main(String[] args) in c:\Users\Josh Keegan\Docume
nts\Visual Studio 2013\Projects\ReproduceBug130\ReproduceBug130\Program.cs:line
121

Cheers,
Josh
Coordinator
Apr 18, 2015 at 11:36 AM
Edited Apr 18, 2015 at 11:39 AM
Hi,

Sorry for the long delays. As I said, I am quite busy right now.
Please try the new version of Tweetinvi in which I added a fix for your specific issue.
This might not fix your problem so please let me know.

https://www.codeplex.com/Download/AttachmentDownload.ashx?ProjectName=tweetinvi&WorkItemId=2546&FileAttachmentId=3381

Linvi
Apr 19, 2015 at 11:23 AM
Hi Linvi,
I appreciate you're a busy man, so thank you very much for taking the time to look into this!

I've been running the program to reproduce the bug with your fix for the past ~20 hrs and it has been running much better than before.
There are still some timeouts, but they may be genuine (Twitter actually timing out).
I'll improve the logging to make sure the timeouts are not consistently for a single user, or anything else which might indicate some sort of problem with TweetInvi rather than Twitter and then get back to you.

One thing I did notice was that TweetinviConfig.CURRENT_WEB_REQUEST_TIMEOUT is still getting ignored by the /1.1/application/rate_limit_status.json requests, which timeout after 10s.
Thanks,
Josh
Coordinator
Apr 20, 2015 at 12:11 AM
Hi,

I will verify the Timeout issue. I have created a new bug related with this.

https://tweetinvi.codeplex.com/workitem/2552

Linvi
Apr 20, 2015 at 2:51 PM
Hi Linvi,
I've switched the live code over to your Beta version today and it is running much better.
Timeouts are almost non-existent when running on the server, so the few timeouts I was seeing when running the Bug Reproducer program were probably caused by my (comparatively) measly internet connection.

There have been some JSON deserialization errors that I have not previously been having though:
2015-04-20 14:42:24 - TwitterRestPoller.Program: Error: Exception whilst fetching User Mentions. Exception:
Newtonsoft.Json.JsonSerializationException: Cannot deserialize the current JSON object (e.g. {"name":"value"}) into type 'System.Collections.Generic.IEnumerable`1[Tweetinvi.Core.Interfaces.DTO.ITweetDTO]' because the type requires a JSON array (e.g. [1,2,3]) to deserialize correctly.
To fix this error either change the JSON to a JSON array (e.g. [1,2,3]) or change the deserialized type so that it is a normal .NET type (e.g. not a primitive type like integer, not a collection type like an array or List<T>) that can be deserialized from a JSON object. JsonObjectAttribute can also be added to the type to force it to deserialize from a JSON object.
Path 'errors', line 1, position 10.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonConverter[] converters)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonConverter[] converters)
   at Tweetinvi.Logic.Wrapper.JsonConvertWrapper.DeserializeObject[T](String json, JsonConverter[] converters)
   at Tweetinvi.Logic.JsonConverters.JsonObjectConverter.DeserializeObject[T](String json, JsonConverter[] converters)
   at Tweetinvi.Credentials.TwitterAccessor.ExecuteGETQuery[T](String query, JsonConverter[] converters)
   at Tweetinvi.Controllers.Timeline.TimelineQueryExecutor.GetMentionsTimeline(IMentionsTimelineRequestParameters timelineRequestParameters)
   at Tweetinvi.Controllers.Timeline.TimelineController.GetMentionsTimeline(IMentionsTimelineRequestParameters mentionsTimelineRequestParameters)
   at Tweetinvi.Controllers.Timeline.TimelineController.GetMentionsTimeline(Int32 maximumNumberOfTweets)
   at Tweetinvi.Timeline.GetMentionsTimeline(Int32 maximumTweets)
   at TwitterRestPoller.Program.fetchUserMentions(TwitterAccount user) in c:\Users\Josh Keegan\Documents\Visual Studio 2013\Projects\CommsManager\TwitterRestPoller\Program.cs:line 430
   at TwitterRestPoller.Program.pollUserMentions(TwitterAccount user) in c:\Users\Josh Keegan\Documents\Visual Studio 2013\Projects\CommsManager\TwitterRestPoller\Program.cs:line 413
These aren't happening all of the time, but have been happening occasionally (so far) for ITweetDTO and IMessageDTO.
Thanks,
Josh
Coordinator
Apr 20, 2015 at 3:50 PM
Thanks for the feedback. I will definitely have a look into it before releasing the next version.

Linvi
Coordinator
Apr 20, 2015 at 4:21 PM
Hi,

Could you please give me the arguments used to get UserMentions when the bug happens.
I cannot reproduce it on my machine.

If possible give me the 2 following information : json + query.
TweetinviEvents.QueryAfterExecute += (sender, args) =>
{
    var jsonResult = args.JsonResult;
    var query = args.QueryURL;
};
Thanks for your help
Apr 21, 2015 at 2:19 PM
Hi Linvi,
I'm having some trouble getting you that information because that event doesn't seem to be firing for every type of query.
I'd left it running overnight and have got a number of exceptions like the one I posted yesterday, but when searching the Logs DB for distinct query URLs I only get:
Query: https://api.twitter.com/1.1/application/rate_limit_status.json
Query: https://api.twitter.com/1.1/direct_messages.json?count=200

I've changed it to try and find the problem in the direct messages with:
TweetinviEvents.QueryAfterExecute += (sender, args) =>
            {
                var jsonResult = args.JsonResult;
                var query = args.QueryURL;

                DefaultLog.Debug("Query: {0}", query);
                DefaultLog.Debug("JSON Result:\n{0}", jsonResult);

                if(query.Contains("direct_messages.json"))
                {
                    try
                    {
                        IMessageDTO messageDto = Newtonsoft.Json.JsonConvert.DeserializeObject<MessageDTO>(jsonResult);
                    }
                    catch(Exception e)
                    {
                        DefaultLog.Error("MessageDTO Derserialization Exception:\n{0}", e);
                    }
                }
            };
Hopefully that should throw something up.
Cheers,
Josh
Coordinator
Apr 21, 2015 at 2:57 PM
Hi,

I wanted to let you know that I could not identify a problem with the Timeout property.
Could you please verify that the operation is performed in the same thread as the thread in which you set the property?

Linvi
Coordinator
Apr 21, 2015 at 3:14 PM
Hi,

The only WebRequest that should not raise this event should normally be the automatic requests for RateLimit when using the RateLimitAwaiter.
I verified the code and I cannot see any way for a WebRequest to be executed without the event to be raised.

But your additional code could be great to find out failure for deserializing Messages.

Also, I wanted to know if you could share your code with me (privately) so that we can work together on these problems.
The reason is that I would like to accelerate the bug fixing of these 2 issues (more specifically the serialization) because its the only remaining issue before releasing Tweetinvi 0.9.7.0.

Cheers,
Linvi
Apr 21, 2015 at 3:26 PM
Hi Linvi,
I've just got a deserialization error and the JSON was just []
I'll leave that debug code in overnight and see if there is anything other than [] causing exceptions.
I assume this is Twitters fault, unless you've used [] as a default anywhere?


As for the timeouts, I'd just added:
TweetinviConfig.CURRENT_WEB_REQUEST_TIMEOUT = 15000;
to the program attached to the original post, at the top of the setThreadCredentials method.
Unless I've missed something this should be in the same thread. Here's it happening:
Image
Regards,
Josh Keegan
Coordinator
Apr 21, 2015 at 4:00 PM
I did not notice but you try to deserialize into a MessageDTO instead of a MessageDTO[].
The error will always occur when performing the get messages request.

Linvi
Apr 21, 2015 at 4:40 PM
Thanks, that wasn't too bright of me!
I've changed it now.

I was surprised that I hadn't been inundated with error messages because of this, but the event hasn't fired for over an hour now.
However, the log is full of messages saying private messages were successfully received. The logging is done after the messages have been fetched, but before doing any processing:
if (messages != null)
{
    DefaultLog.Debug("Fetched Direct Messages (Outbound) for user {0}. Inserting or updating them in the database", user.DisplayName);
    InboundPrivateMessage.Process(messages);
 }
So something isn't quite right.

I've assumed that the QueryAfterExecute event isn't thread specific, is that right?
Is there anything else that would stop the event from firing?
Cheers,
Josh
Apr 28, 2015 at 7:17 AM
Its working if query is run in the background
Private Sub MainWindow_Loaded(ByVal sender As Object, ByVal e As System.Windows.RoutedEventArgs) Handles Me.Loaded
Dim bg As New BackgroundWorker
        AddHandler bg.DoWork, AddressOf bw_DoWork
        bg.RunWorkerAsync()
End Sub

Private Sub bw_DoWork(ByVal sender As Object, ByVal e As DoWorkEventArgs)
        Dim tweets = Search.SearchTweets("#obama")
        For Each tweet As ITweet In tweets
             Msgbox(tweet.text)
        Next
                                       
End Sub
Coordinator
Apr 28, 2015 at 9:00 AM
Hi Amit,

Could please confirm the version of Tweetinvi you are using?

Cheers,
Linvi
Apr 28, 2015 at 9:04 AM
Edited Apr 28, 2015 at 9:12 AM
0.9.7.0

The problem is that main thread is getting blocked while executing the query and main thread time out after 10sec and throws the exception. So all queries should be run in background in wpf application. In asp.net, it works perfectly well.
Apr 28, 2015 at 9:38 AM
Hi Amit,
Linvi included a fix for this problem in TweetInvi 0.9.7.0.
Cheers,
Josh
Marked as answer by linvi on 4/29/2015 at 1:54 AM