Web application help

Nov 7, 2014 at 1:54 PM
I am trying to make a web application in c# that will allow me to search for tweets based on a keyword in the application and then return the number of tweets that contain this keyword.
Does anybody have any advice on how to do this?
Nov 7, 2014 at 3:28 PM
Hi,

I've already done the exact same application in the past. What you want to use is the FilteredStream.
The FilteredStream is simply getting all the tweets matching a specific set of criteria.

Each time you receive a Tweet, you will need to store it in a Database. Your website will then only need to query the Database to get the information it wants.
I would advise that you save the Tweet in the database as a set of 100 to 500 tweets in order for the Database to follow the speed requested to add your tweets.

You do not want to use the Search because it is Rate limited and you'll probably not be able to get all the tweets from it.

Linvi
Nov 7, 2014 at 5:58 PM
Thanks for your response.
I don't suppose you have an example of this application available anywhere?
It would really save me.
Nov 7, 2014 at 6:33 PM
Also I would just like to emphasize that I'm not looking for the app to return the tweets themselves but to return the amount of tweets,
for example: the keyword Manchester united has been mentioned 123 times.
would this change anything or would I still be using the filtered stream?
Nov 7, 2014 at 6:51 PM
Unfortunately, I will not be able to share the code with you as the project has been done for a client.
But this is a simple matter of storing the number of Tweets somewhere (file/database) and displaying it.

The basic following code should work for you:
var fs = Stream.CreateFilteredStream();
fs.AddTrack("Manchester");
fs.MatchingTweetReceived += (sender, args) =>
{
    Console.WriteLine("Tweet with manchester has been found +1 the database!");
};
fs.StartStreamMatchingAnyCondition();
Obviously you will have to change the Console.WriteLine into the code to change the value in the database/file.

If you need any more help.
Please let me know.

Regards,
Linvi
Nov 7, 2014 at 6:55 PM
I keep getting this error in relation to sender, do you know what it means?

Error 1 A local variable named 'sender' cannot be declared in this scope because it would give a different meaning to 'sender', which is already used in a 'parent or current' scope to denote something else C:\Users\steph_000\Documents\Visual Studio 2013\Projects\AppPrototype\AppPrototype\Home.aspx.cs 30 34 AppPrototype
Nov 7, 2014 at 7:23 PM
Edited Nov 7, 2014 at 7:26 PM
Hi,

It just informs you that the sender variable already exist in a your code scope. Simply replace sender by s.
fs.MatchingTweetReceived += (s, args) => { };
Linvi
Nov 7, 2014 at 7:33 PM
Okay here's my code for the Home.aspx.cs page:


using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Threading;
using Tweetinvi;
using Tweetinvi.Core.Extensions;
using Tweetinvi.Core.Interfaces.oAuth;
using Tweetinvi.Core.Interfaces.Streaminvi;
using Tweetinvi.Json;
using Geo = Tweetinvi.Geo;
using SavedSearch = Tweetinvi.SavedSearch;
using Stream = Tweetinvi.Stream;

namespace AppPrototype
{
public partial class Home : System.Web.UI.Page
{
    protected void Page_Load(object sender, EventArgs e)
    {
        var fs = Stream.CreateFilteredStream();

        fs.AddTrack("Manchester United");
        fs.MatchingTweetReceived += (s, args) =>
            {

                Console.WriteLine("The number of tweets with manchester has been found x times in the database");

            };

        fs.StartStreamMatchingAnyCondition();

    }
}
}

And here's my code for the web form/Home.aspx page that the information is being pulled into:

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="Home.aspx.cs" Inherits="AppPrototype.Home" %>

<!DOCTYPE html>

<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server"> <title>Twitter Keyword Search</title> </head> <body>
<form id="form1" runat="server">
<div>

    <asp:Label ID="keywordSearch" runat="server" Text=""></asp:Label>
    <br />
    <br />
    <asp:Label ID="successlbl" runat="server" Text=""></asp:Label>
    <br />
    <br />
    <asp:Label ID="resultlbl" runat="server" Text=""></asp:Label>
    <br />




</div>
</form>
</body>
</html>


Can you see anything wrong?
Again thank you for all your help, You're really saving my life now
Kind regards
Stephen
Nov 7, 2014 at 8:30 PM
Hi,

You have 2 problems that I can see here. The first one is that you try to access a stream from a webpage.
I think you should have this done on your server as webservice because it is an ongoing process.

Second you need to specify credentials when you want to use the Twitter API.
To get such credentials you need to go on https://apps.twitter.com/.

Then in Tweetinvi you need to specify the credentials as the first operation you do with Tweetinvi in your code.
TwitterCredentials.SetCredentials("Access_Token", "Access_Token_Secret", "Consumer_Key", "Consumer_Secret");
I hope this makes some sense.
Linvi
Nov 7, 2014 at 8:54 PM
Hi

This is great, you have been a huge help thank you.
I have one finally question then the last time you hear from me will be a mention in my graduation speech.
You mentioned earlier that I would need to change the console.writeline into code that will let me change
values in my connected database. I am not very familiar with databases and I am not sure how to do this,
would you have an example of this code?
Once again I cannot thank you enough.

regards
Stephen
Nov 8, 2014 at 2:36 PM
Hi,

Yes, this is very easy. As I said you can either use a File as a CSV or a database. This link describes how to use the database easily with Visual Studio 2012.

To create the table :
CREATE TABLE [dbo].[NumberOfKeywords](
    [Id] [int] NOT NULL,
    [Keyword] [nvarchar](MAX) NOT NULL,
    [NumberOfKeywords] [int] NOT NULL,
 CONSTRAINT [PK_NumberOfKeywords] PRIMARY KEY CLUSTERED 
(
    [Id] ASC
)
WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
This will allow you to link keywords to a number of time they have been found.
Then as explained use the DataSet to store and access the information stored.

Regards,
Linvi
Nov 10, 2014 at 12:38 PM
Hi

I am having a bit of trouble with the application, its connecting to twitter and getting the results from the keyword but its not sending these results to the databases would you have any idea as to whats wrong?
Nov 10, 2014 at 12:57 PM
Hi,

It depends on which technology you use to update your database. Without your code I won't be able to help you.

Linvi
Nov 10, 2014 at 7:29 PM
Hi

I'm sorry I wasn't able to respond sooner but I have another issue with the project.
I have been told that a database would be unsuitable to work into this project.
Would you know of any way to pull the information into something like a label?
Should I still use the filtered stream to go about getting the counted instances of tweets containing my keyword or should I use something else?
Thanks for your help.

Stephen
Nov 11, 2014 at 4:25 PM
Hi,

I have been told that a database would be unsuitable to work into this project.

The only thing you need is a file or whatever container that will store the information you need.
You don't need this to be a database as I explained, but DB are done for this purpose.

Have a look here for reading and updating a file.
// Reading the number
string numberAsString = File.ReadAllText(fileLocation);
int number = Int32.Parse(numberAsString);

// Updating the number
int number = 42;
File.WriteAllText(fileLocation, number.ToString(CultureInfo.InvariantCulture));
Would you know of any way to pull the information into something like a label?

I do not really understand the question :p

Should I still use the filtered stream to go about getting the counted instances of tweets containing my keyword or should I use something else?

In any case I strongly advise to use the FilteredStream as any other solution will be rate limited and more complicated to implement.

Linvi
Nov 11, 2014 at 4:55 PM
Edited Nov 11, 2014 at 4:57 PM
Hi

When I said could I pull the information into a label I meant would it be possible to display the data I receive from the filtered stream in a <asp:Label like the way I can do this using the search function, for example: newlbl.Text = thirdTweet.ToString();.
I am having difficulty displaying the information that I am receiving from the filtered stream, Would you know a way that I could just pull the number of times my keyword has been mentioned directly from twitter and display this information as an int?, i.e the number of tweets containing Manchester united is 419 and displaying just this number.

I have been told that trying to get my app to count every tweet containing my keyword would be unsuitable due to the fact that it would have to count thousands of tweets and this will take too much time and memory.

Thanks for your help
Stephen

PS. I was having issues with my codeplex account so I had to use my friends, that's why the names are different
Nov 11, 2014 at 5:54 PM
Edited Nov 11, 2014 at 5:59 PM
Hi,

What is a FilteredStream and how to use it?

I am not sure you understand what the FilteredStream is and what it does. A FilteredStream is a stream provided by Twitter that runs 24/24 that returns all the tweets matching a specific set of criteria.

Because Twitter does not provide any powerfull Search API you must use this in order to count the number of tweets that have been published. Therefore, you will need to have a Thread/Process, running 24/24 responsible to run the FilteredStream and store the number of tweets that matched your criteria.

In the same time, you want to display information to the end user. This cannot be done in the same thread as it is already busy getting information from Twitter.
Trying to store and display information in the same thread will therefore result in the page loading indefinitely.

Why a file/database?

Now you have 2 threads operating independently. A first one in a console for example storing the data. And a second one an IIS Server (lets not talk about the ThreadPools) distributing the Views/Pages of your ASP.NET Website.

A question comes to mind, how can we communicate the data between these 2 threads/process? Well the simples solution and mostly used is a Database. A database being a file, we could also simply use a file that both threads/processes can access. If the first process modifies the data of the file, you will be able to read it the next time you load a page in your IIS Server.

The LABEL

When you have your IIS running it should be very easy to display the label. You will simply need to access the data from the file in your code behind and then display it in the View. This is the only thing you should need to do in the WebServer/Website.

I have been told that trying to get my app to count every tweet containing my keyword would be unsuitable due to the fact that it would have to count thousands of tweets and this will take too much time and memory.

This is totally false. By simply using 2 different threads the time will be of 10 ms to start the Thread. After that, well it is a thread so it will always run but this is the goal. If it did not run constantly you would not be able to get the information you want. Concerning the memory, I have stored more than 100 fully loaded (with all data not only the number of tweets) during months without running in any kind of problem.

PS

I am trying to help you, but you need to understand the concept of multiple threads communicating together as well as how to store and access data. I hope this posts makes things easier to understand.

Regards,
Linvi
Nov 11, 2014 at 6:05 PM
Hi

Thanks for your post I think I understand what I need to do, I'll head back to my team and inform them about what you have told me,
I understand that you are trying to help me and you have been a huge help to me and I apologies if I appear ungrateful it's just that I am under a lot of pressure to get this section done.
Once again thanks for your help

Stephen
Nov 11, 2014 at 7:19 PM
Hi

Would It be possible for you to give me a quick walk-through of what i will need to implement this and how I should implement it?

Thanks
Stephen
Nov 13, 2014 at 12:53 AM
Hi,

Please let me know what you mean by walk-through? I cannot think of any information I did not gave you.

Linvi
Nov 13, 2014 at 11:23 AM
Hi

When I say walk-through I mean like step 1 create the page congaing the keywords, step 2 create a file to store the information,
things like that.
Thanks
Stephen