Adding keywords to stream

Apr 29, 2015 at 10:28 PM
Edited Apr 29, 2015 at 10:46 PM
Hi,

First of all, thanks for your work on this library. It seems great and works perfectly for my purposes.

That said I have a question regarding adding new keywords to an existing stream which I couldn't figure out. I read a few other discussions regarding this point but didn't manage to get the answers working. Everything works for me until I need to change the keywords being tracked.

If I do something like the following I would guess that the stream would start and send me tweets that match my initial phrase. After 30s, I would like to add on a new, higher volume keyword. This does not work however, the stream does restart but the new phrase doesn't take effect. I also get the impression even the old one sends fewer hits than before the timer hits. What am I doing wrong? Should I be disposing of the stream and starting a completely new one instead?

Also, is the StartStreamMatchingAllConditions() supposed to be blocking? Sometimes when running it in the debugger it doesn't block and I can't figure out if it's the debugger messing with me or if it's an actual problem.

Thank you very much in advance!


filteredStream.AddTrack("a low volume phrase");
filteredStream.JsonObjectReceived += filteredStream_JsonObjectReceived;

var resetTimer = new System.Timers.Timer(30000);
resetTimer.Elapsed += (o, eventArgs) =>
            {
                filteredStream.StopStream();
        //Add a high volume keyword
        filteredStream.AddTrack("lady gaga");
                filteredStream.StartStreamMatchingAllConditions();
            };
resetTimer.Start();
filteredStream.StartStreamMatchingAllConditions();
Apr 30, 2015 at 1:59 AM
Hi Karl,

Thank you very much for your generous donation. This is much appreciated.

Concerning your issue, you are on the right track, the stream needs to be stopped in order to add a track.
Though you need to understand how the filter string works.

https://dev.twitter.com/streaming/overview/request-parameters#track

To make it simple, when you ask for "lady gaga", you ask for a tweet that contains both "lady" AND "gaga" as 2 separate words.
If you want to use an OR you need to add 2 tracks.
// "lady" AND "gaga"
filteredStream.AddTrack("lady gaga");

// "lady" OR "gaga"
filteredStream.AddTrack("lady");
filteredStream.AddTrack("gaga");
Please let me know if this helps you to solve your problem.
Thanks again for your donation :)
Linvi
Apr 30, 2015 at 7:15 AM
Yes, I understand that the API works in AND mode. This is not my question. If I create a new stream and
filteredStream.AddTrack("lady gaga");
filteredStream.StartStreamMatchingAllConditions();
It works as expected and tweets start coming in quite quickly since "lady gaga" is a high volume keyword. However, if I I want to stop, add a new keyword and then restart the stream the added keyword does not seem to take effect:
//Add a keyword that won't match anything
filteredStream.AddTrack("gdfkjghsfgdsfg");
filteredStream.StartStreamMatchingAllConditions();

//Then, from a timer event handler,
filteredStream.StopStream();
filteredStream.AddTrack("lady gaga");
filteredStream.StartStreamMatchingAllConditions();
After I restart the stream I will still not get any results.
Apr 30, 2015 at 9:39 AM
Hi,

I will investigate this and get back to you when this is done. It might be at the end of the day.

Regards,
Linvi
Apr 30, 2015 at 10:46 AM
What you encounter is a threading bug that I have introduced when implementing the async framework in Streams.

As a work around from this bug (on which I will work as a priority), please add the following code to yours.
filteredStream.StreamStopped += (sender, args) =>
{
           filteredStream.StartStreamMatchingAllConditions();
};
I know that this is not ideal but the bug is not trivial so I will have to think about this before releasing a fix.

Work Item related with Bug

Sorry for the inconvenience and thank you for reporting this.
Linvi
May 1, 2015 at 1:44 AM
Hi Karl,

I wanted to let you know that the issue you encountered should be fixed in the latest Source Code version.
If you want I can send you the DLL, or you can build the project with Visual Studio.

Please give me some feedback.
Cheers,
Linvi
Marked as answer by linvi on 4/30/2015 at 5:44 PM
May 3, 2015 at 8:37 PM
Hi,

I still had problems but I finally got it working. The problem was that I was calling
filteredStream.Tracks.Clear();
Instead of
filteredStream.ClearTracks();
Doing it the first way breaks the stream in the same way as the previous example without any obvious reason. I assume this is threading-related as well and I'm not supposed to go clearing data structures myself :)

Thanks again for your help!
May 3, 2015 at 10:16 PM
Actually I think I might have spoken too soon. It works most of the time but periodically it just stops working when changing keywords. I'm not really sure where to start looking into this problem. I can trigger it pretty consistently by adding and removing keywords a few times.

I noticed that
filteredStream.StartStreamMatchingAllConditions();
returns (i.e. stops blocking) if StopStream is called from a different thread, and then if you look at GetLastException() you get a NullReferenceException. I'm not sure if this is normal or not.

Do you have any example code regarding how best to structure a program that listens to the stream for a long time? Right now I'm spinning off a thread that starts filteredStream.StartStreamMatchingAllConditions(); but since that returns whenever the stream is stopped it's a little difficult to determine the status of the stream (it might have been restarted later, like in the timer example above).

Sorry for all the questions. It's probably my inexperience in multithreading showing through...
May 4, 2015 at 11:48 PM
Hi,

I have attached the latest version of the binaries to the issue. You can download them with the following link.

https://www.codeplex.com/Download/AttachmentDownload.ashx?ProjectName=tweetinvi&WorkItemId=2566&FileAttachmentId=3399

Cheers,
Linvi